Note: This is test shard 1 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AdminCliTest
[ RUN ] AdminCliTest.RebalancerReportOnly
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 01:52:28.812671 15187 test_util.cc:268] Using random seed: -620260019
I20240502 01:52:28.824950 15187 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:52:28.825044 15187 ts_itest-base.cc:113] --------------
I20240502 01:52:28.825088 15187 ts_itest-base.cc:114] 5 tablet servers
I20240502 01:52:28.825126 15187 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:52:28.825192 15187 ts_itest-base.cc:116] --------------
2024-05-02T01:52:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:52:28Z Disabled control of system clock
I20240502 01:52:28.851414 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.254:33883
--webserver_interface=127.14.212.254
--webserver_port=0
--builtin_ntp_servers=127.14.212.212:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.254:33883 with env {}
W20240502 01:52:29.192363 15199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:29.192711 15199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:29.192806 15199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:29.200575 15199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:52:29.200713 15199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:29.200767 15199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:52:29.200858 15199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:52:29.210400 15199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.254:33883
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.254:33883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.212.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:29.211817 15199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:29.213631 15199 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:29.225970 15205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:29.226518 15207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:29.227154 15204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:29.228137 15199 server_base.cc:1034] running on GCE node
I20240502 01:52:29.229055 15199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:29.230573 15199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:29.231807 15199 hybrid_clock.cc:648] HybridClock initialized: now 1714614749231737 us; error 47 us; skew 500 ppm
I20240502 01:52:29.232138 15199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:29.234591 15199 webserver.cc:460] Webserver started at http://127.14.212.254:46539/ using document root <none> and password file <none>
I20240502 01:52:29.235337 15199 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:29.235478 15199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:29.235797 15199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:29.238647 15199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "eb2b9ad2a55644cab40dba52bda66a18"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-mk1r"
I20240502 01:52:29.239423 15199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "eb2b9ad2a55644cab40dba52bda66a18"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-mk1r"
I20240502 01:52:29.245049 15199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 01:52:29.248651 15213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:29.250066 15199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:29.250239 15199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
uuid: "eb2b9ad2a55644cab40dba52bda66a18"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-mk1r"
I20240502 01:52:29.250432 15199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:29.265609 15199 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:29.266510 15199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:29.266835 15199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:29.286067 15199 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.254:33883
I20240502 01:52:29.286140 15264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.254:33883 every 8 connection(s)
I20240502 01:52:29.287907 15199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:52:29.290365 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 15199
I20240502 01:52:29.290692 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:52:29.293331 15265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:29.306094 15265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18: Bootstrap starting.
I20240502 01:52:29.309746 15265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:29.311048 15265 log.cc:824] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:29.314481 15265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18: No bootstrap required, opened a new log
I20240502 01:52:29.323238 15265 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2b9ad2a55644cab40dba52bda66a18" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 33883 } }
I20240502 01:52:29.323624 15265 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:29.323735 15265 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2b9ad2a55644cab40dba52bda66a18, State: Initialized, Role: FOLLOWER
I20240502 01:52:29.324388 15265 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2b9ad2a55644cab40dba52bda66a18" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 33883 } }
I20240502 01:52:29.324612 15265 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:29.324755 15265 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:29.324959 15265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:29.327188 15265 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2b9ad2a55644cab40dba52bda66a18" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 33883 } }
I20240502 01:52:29.327718 15265 leader_election.cc:302] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb2b9ad2a55644cab40dba52bda66a18; no voters:
I20240502 01:52:29.328395 15265 leader_election.cc:288] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:29.328548 15270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:29.330425 15270 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18 [term 1 LEADER]: Becoming Leader. State: Replica: eb2b9ad2a55644cab40dba52bda66a18, State: Running, Role: LEADER
I20240502 01:52:29.331079 15265 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:29.331151 15270 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2b9ad2a55644cab40dba52bda66a18" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 33883 } }
I20240502 01:52:29.334422 15271 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb2b9ad2a55644cab40dba52bda66a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2b9ad2a55644cab40dba52bda66a18" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 33883 } } }
I20240502 01:52:29.334415 15272 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb2b9ad2a55644cab40dba52bda66a18. Latest consensus state: current_term: 1 leader_uuid: "eb2b9ad2a55644cab40dba52bda66a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2b9ad2a55644cab40dba52bda66a18" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 33883 } } }
I20240502 01:52:29.334920 15271 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:29.335012 15272 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:29.342649 15279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:52:29.349184 15279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:52:29.358637 15279 catalog_manager.cc:1347] Generated new cluster ID: 9c20997498da46079b9a9c17dc533b1a
I20240502 01:52:29.358772 15279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:52:29.374744 15279 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:52:29.376551 15279 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:52:29.398877 15279 catalog_manager.cc:5857] T 00000000000000000000000000000000 P eb2b9ad2a55644cab40dba52bda66a18: Generated new TSK 0
I20240502 01:52:29.399777 15279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:52:29.411471 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.193:0
--local_ip_for_outbound_sockets=127.14.212.193
--webserver_interface=127.14.212.193
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33883
--builtin_ntp_servers=127.14.212.212:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240502 01:52:29.763329 15289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:29.763660 15289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:29.763780 15289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:29.771848 15289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:29.772063 15289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.193
I20240502 01:52:29.782810 15289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.212.193
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33883
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.193
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:29.784210 15289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:29.786082 15289 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:29.798856 15294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:29.800071 15295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:29.800637 15297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:29.801800 15289 server_base.cc:1034] running on GCE node
I20240502 01:52:29.802492 15289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:29.803539 15289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:29.804795 15289 hybrid_clock.cc:648] HybridClock initialized: now 1714614749804708 us; error 55 us; skew 500 ppm
I20240502 01:52:29.805143 15289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:29.811718 15289 webserver.cc:460] Webserver started at http://127.14.212.193:37719/ using document root <none> and password file <none>
I20240502 01:52:29.812640 15289 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:29.812796 15289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:29.813150 15289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:29.815977 15289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d7dad9f368e747e192ca2e21396cb1da"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-mk1r"
I20240502 01:52:29.816745 15289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d7dad9f368e747e192ca2e21396cb1da"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-mk1r"
I20240502 01:52:29.822494 15289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240502 01:52:29.826041 15304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:29.827419 15289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:52:29.827580 15289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d7dad9f368e747e192ca2e21396cb1da"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-mk1r"
I20240502 01:52:29.827761 15289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:29.844751 15289 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:29.845643 15289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:29.845944 15289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:29.846921 15289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:29.848448 15289 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:29.848529 15289 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:29.848663 15289 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:29.848719 15289 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:29.882936 15289 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.193:34549
I20240502 01:52:29.882975 15416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.193:34549 every 8 connection(s)
I20240502 01:52:29.884351 15289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:52:29.893920 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 15289
I20240502 01:52:29.894177 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:52:29.898267 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.194:0
--local_ip_for_outbound_sockets=127.14.212.194
--webserver_interface=127.14.212.194
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33883
--builtin_ntp_servers=127.14.212.212:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 01:52:29.898752 15417 heartbeater.cc:347] Connected to a master server at 127.14.212.254:33883
I20240502 01:52:29.899367 15417 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:29.900529 15417 heartbeater.cc:511] Master 127.14.212.254:33883 requested a full tablet report, sending...
I20240502 01:52:29.903065 15230 ts_manager.cc:194] Registered new tserver with Master: d7dad9f368e747e192ca2e21396cb1da (127.14.212.193:34549)
I20240502 01:52:29.904592 15230 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.193:60441
W20240502 01:52:30.238848 15421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:30.239199 15421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:30.239351 15421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:30.247481 15421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:30.247731 15421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.194
I20240502 01:52:30.257295 15421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.212.194
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33883
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.194
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:30.258745 15421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:30.260583 15421 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:30.274134 15429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:30.274976 15421 server_base.cc:1034] running on GCE node
W20240502 01:52:30.275630 15426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:30.276237 15427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:30.277172 15421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:30.278295 15421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:30.279554 15421 hybrid_clock.cc:648] HybridClock initialized: now 1714614750279467 us; error 57 us; skew 500 ppm
I20240502 01:52:30.279917 15421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:30.282580 15421 webserver.cc:460] Webserver started at http://127.14.212.194:37615/ using document root <none> and password file <none>
I20240502 01:52:30.283411 15421 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:30.283555 15421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:30.283896 15421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:30.286826 15421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c35e5f91cdd44178b1a78e6c9838563e"
format_stamp: "Formatted at 2024-05-02 01:52:30 on dist-test-slave-mk1r"
I20240502 01:52:30.287614 15421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c35e5f91cdd44178b1a78e6c9838563e"
format_stamp: "Formatted at 2024-05-02 01:52:30 on dist-test-slave-mk1r"
I20240502 01:52:30.293417 15421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240502 01:52:30.297132 15436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:30.298704 15421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:52:30.298873 15421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c35e5f91cdd44178b1a78e6c9838563e"
format_stamp: "Formatted at 2024-05-02 01:52:30 on dist-test-slave-mk1r"
I20240502 01:52:30.299068 15421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:30.337545 15421 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:30.338485 15421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:30.338796 15421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:30.339855 15421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:30.341486 15421 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:30.341569 15421 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:30.341708 15421 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:30.341779 15421 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:30.376411 15421 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.194:41377
I20240502 01:52:30.376470 15549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.194:41377 every 8 connection(s)
I20240502 01:52:30.377841 15421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:52:30.378918 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 15421
I20240502 01:52:30.379117 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:52:30.383322 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.195:0
--local_ip_for_outbound_sockets=127.14.212.195
--webserver_interface=127.14.212.195
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33883
--builtin_ntp_servers=127.14.212.212:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 01:52:30.395205 15550 heartbeater.cc:347] Connected to a master server at 127.14.212.254:33883
I20240502 01:52:30.395537 15550 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:30.396467 15550 heartbeater.cc:511] Master 127.14.212.254:33883 requested a full tablet report, sending...
I20240502 01:52:30.398038 15230 ts_manager.cc:194] Registered new tserver with Master: c35e5f91cdd44178b1a78e6c9838563e (127.14.212.194:41377)
I20240502 01:52:30.398845 15230 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.194:49299
W20240502 01:52:30.725428 15553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:30.725771 15553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:30.725916 15553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:30.733852 15553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:30.734077 15553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.195
I20240502 01:52:30.743495 15553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.212.195
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33883
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.195
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:30.744859 15553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:30.746709 15553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:30.759364 15558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:30.760139 15559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:30.760700 15561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:30.761454 15553 server_base.cc:1034] running on GCE node
I20240502 01:52:30.762130 15553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:30.763124 15553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:30.764340 15553 hybrid_clock.cc:648] HybridClock initialized: now 1714614750764274 us; error 44 us; skew 500 ppm
I20240502 01:52:30.764639 15553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:30.766960 15553 webserver.cc:460] Webserver started at http://127.14.212.195:42519/ using document root <none> and password file <none>
I20240502 01:52:30.767689 15553 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:30.767838 15553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:30.768149 15553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:30.770951 15553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "29fba4a87cd543ff99c483f497344b98"
format_stamp: "Formatted at 2024-05-02 01:52:30 on dist-test-slave-mk1r"
I20240502 01:52:30.771692 15553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "29fba4a87cd543ff99c483f497344b98"
format_stamp: "Formatted at 2024-05-02 01:52:30 on dist-test-slave-mk1r"
I20240502 01:52:30.777082 15553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20240502 01:52:30.780761 15567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:30.782282 15553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:30.782450 15553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "29fba4a87cd543ff99c483f497344b98"
format_stamp: "Formatted at 2024-05-02 01:52:30 on dist-test-slave-mk1r"
I20240502 01:52:30.782683 15553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:30.804566 15553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:30.805473 15553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:30.805778 15553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:30.806752 15553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:30.808336 15553 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:30.808418 15553 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:30.808553 15553 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:30.808612 15553 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:30.850229 15553 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.195:42229
I20240502 01:52:30.850273 15681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.195:42229 every 8 connection(s)
I20240502 01:52:30.851606 15553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:52:30.853346 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 15553
I20240502 01:52:30.853621 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:52:30.857332 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.196:0
--local_ip_for_outbound_sockets=127.14.212.196
--webserver_interface=127.14.212.196
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33883
--builtin_ntp_servers=127.14.212.212:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 01:52:30.867357 15682 heartbeater.cc:347] Connected to a master server at 127.14.212.254:33883
I20240502 01:52:30.867669 15682 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:30.868587 15682 heartbeater.cc:511] Master 127.14.212.254:33883 requested a full tablet report, sending...
I20240502 01:52:30.870222 15230 ts_manager.cc:194] Registered new tserver with Master: 29fba4a87cd543ff99c483f497344b98 (127.14.212.195:42229)
I20240502 01:52:30.871017 15230 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.195:41955
I20240502 01:52:30.907686 15417 heartbeater.cc:503] Master 127.14.212.254:33883 was elected leader, sending a full tablet report...
W20240502 01:52:31.193092 15685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:31.193424 15685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:31.193576 15685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:31.201419 15685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:31.201644 15685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.196
I20240502 01:52:31.211001 15685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.212.196
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33883
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.196
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:31.212392 15685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:31.214128 15685 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:31.228703 15691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:31.229211 15693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:31.229776 15690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:31.230829 15685 server_base.cc:1034] running on GCE node
I20240502 01:52:31.231444 15685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:31.232568 15685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:31.233806 15685 hybrid_clock.cc:648] HybridClock initialized: now 1714614751233730 us; error 50 us; skew 500 ppm
I20240502 01:52:31.234131 15685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:31.236608 15685 webserver.cc:460] Webserver started at http://127.14.212.196:33241/ using document root <none> and password file <none>
I20240502 01:52:31.237417 15685 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:31.237553 15685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:31.237864 15685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:31.240489 15685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "135ee4b62c6a443cbbc9e233be847b11"
format_stamp: "Formatted at 2024-05-02 01:52:31 on dist-test-slave-mk1r"
I20240502 01:52:31.241281 15685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "135ee4b62c6a443cbbc9e233be847b11"
format_stamp: "Formatted at 2024-05-02 01:52:31 on dist-test-slave-mk1r"
I20240502 01:52:31.246922 15685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 01:52:31.250458 15699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:31.251827 15685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 01:52:31.251991 15685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "135ee4b62c6a443cbbc9e233be847b11"
format_stamp: "Formatted at 2024-05-02 01:52:31 on dist-test-slave-mk1r"
I20240502 01:52:31.252175 15685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:31.298504 15685 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:31.299402 15685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:31.299734 15685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:31.300787 15685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:31.302453 15685 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:31.302538 15685 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:31.302682 15685 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:31.302739 15685 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:31.338308 15685 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.196:43863
I20240502 01:52:31.338352 15813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.196:43863 every 8 connection(s)
I20240502 01:52:31.339601 15685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:52:31.348651 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 15685
I20240502 01:52:31.348867 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:52:31.352978 15814 heartbeater.cc:347] Connected to a master server at 127.14.212.254:33883
I20240502 01:52:31.353228 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.197:0
--local_ip_for_outbound_sockets=127.14.212.197
--webserver_interface=127.14.212.197
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33883
--builtin_ntp_servers=127.14.212.212:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 01:52:31.353402 15814 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:31.354590 15814 heartbeater.cc:511] Master 127.14.212.254:33883 requested a full tablet report, sending...
I20240502 01:52:31.356189 15230 ts_manager.cc:194] Registered new tserver with Master: 135ee4b62c6a443cbbc9e233be847b11 (127.14.212.196:43863)
I20240502 01:52:31.357131 15230 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.196:41709
I20240502 01:52:31.401546 15550 heartbeater.cc:503] Master 127.14.212.254:33883 was elected leader, sending a full tablet report...
W20240502 01:52:31.694574 15817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:31.694887 15817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:31.695016 15817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:31.702782 15817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:31.703013 15817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.197
I20240502 01:52:31.712599 15817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.212.197
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33883
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.197
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:31.714010 15817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:31.715771 15817 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:31.728366 15823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:31.729025 15822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:31.729615 15825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:31.730403 15817 server_base.cc:1034] running on GCE node
I20240502 01:52:31.731114 15817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:31.732160 15817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:31.733379 15817 hybrid_clock.cc:648] HybridClock initialized: now 1714614751733310 us; error 47 us; skew 500 ppm
I20240502 01:52:31.733682 15817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:31.736038 15817 webserver.cc:460] Webserver started at http://127.14.212.197:39633/ using document root <none> and password file <none>
I20240502 01:52:31.736775 15817 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:31.736932 15817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:31.737243 15817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:31.739845 15817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e1ef9f8b5e7f4fac914bed0f9a7b9e02"
format_stamp: "Formatted at 2024-05-02 01:52:31 on dist-test-slave-mk1r"
I20240502 01:52:31.740566 15817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e1ef9f8b5e7f4fac914bed0f9a7b9e02"
format_stamp: "Formatted at 2024-05-02 01:52:31 on dist-test-slave-mk1r"
I20240502 01:52:31.745913 15817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20240502 01:52:31.749414 15831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:31.750841 15817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240502 01:52:31.750995 15817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e1ef9f8b5e7f4fac914bed0f9a7b9e02"
format_stamp: "Formatted at 2024-05-02 01:52:31 on dist-test-slave-mk1r"
I20240502 01:52:31.751171 15817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:31.768057 15817 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:31.768935 15817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:31.769223 15817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:31.770341 15817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:31.771912 15817 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:31.771996 15817 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:31.772130 15817 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:31.772186 15817 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:31.806888 15817 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.197:42221
I20240502 01:52:31.806931 15944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.197:42221 every 8 connection(s)
I20240502 01:52:31.808197 15817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:52:31.812798 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 15817
I20240502 01:52:31.813050 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:52:31.821964 15945 heartbeater.cc:347] Connected to a master server at 127.14.212.254:33883
I20240502 01:52:31.822289 15945 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:31.823259 15945 heartbeater.cc:511] Master 127.14.212.254:33883 requested a full tablet report, sending...
I20240502 01:52:31.824651 15230 ts_manager.cc:194] Registered new tserver with Master: e1ef9f8b5e7f4fac914bed0f9a7b9e02 (127.14.212.197:42221)
I20240502 01:52:31.825549 15230 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.197:60165
I20240502 01:52:31.830163 15187 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:52:31.851599 15230 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35160:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 01:52:31.873693 15682 heartbeater.cc:503] Master 127.14.212.254:33883 was elected leader, sending a full tablet report...
I20240502 01:52:31.904927 15485 tablet_service.cc:1467] Processing CreateTablet for tablet dfbd1032c876475e974ae5b99a5ce28e (DEFAULT_TABLE table=TestTable [id=256c1b7aefea4d25b22523420973c456]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:31.905751 15748 tablet_service.cc:1467] Processing CreateTablet for tablet dfbd1032c876475e974ae5b99a5ce28e (DEFAULT_TABLE table=TestTable [id=256c1b7aefea4d25b22523420973c456]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:31.906786 15485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfbd1032c876475e974ae5b99a5ce28e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:31.906419 15616 tablet_service.cc:1467] Processing CreateTablet for tablet dfbd1032c876475e974ae5b99a5ce28e (DEFAULT_TABLE table=TestTable [id=256c1b7aefea4d25b22523420973c456]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:31.907176 15748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfbd1032c876475e974ae5b99a5ce28e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:31.907872 15616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfbd1032c876475e974ae5b99a5ce28e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:31.918877 15963 tablet_bootstrap.cc:492] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11: Bootstrap starting.
I20240502 01:52:31.921305 15964 tablet_bootstrap.cc:492] T dfbd1032c876475e974ae5b99a5ce28e P c35e5f91cdd44178b1a78e6c9838563e: Bootstrap starting.
I20240502 01:52:31.923103 15965 tablet_bootstrap.cc:492] T dfbd1032c876475e974ae5b99a5ce28e P 29fba4a87cd543ff99c483f497344b98: Bootstrap starting.
I20240502 01:52:31.925182 15964 tablet_bootstrap.cc:654] T dfbd1032c876475e974ae5b99a5ce28e P c35e5f91cdd44178b1a78e6c9838563e: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:31.925781 15963 tablet_bootstrap.cc:654] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:31.925868 15965 tablet_bootstrap.cc:654] T dfbd1032c876475e974ae5b99a5ce28e P 29fba4a87cd543ff99c483f497344b98: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:31.926522 15964 log.cc:824] T dfbd1032c876475e974ae5b99a5ce28e P c35e5f91cdd44178b1a78e6c9838563e: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:31.926788 15965 log.cc:824] T dfbd1032c876475e974ae5b99a5ce28e P 29fba4a87cd543ff99c483f497344b98: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:31.927000 15963 log.cc:824] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:31.929761 15965 tablet_bootstrap.cc:492] T dfbd1032c876475e974ae5b99a5ce28e P 29fba4a87cd543ff99c483f497344b98: No bootstrap required, opened a new log
I20240502 01:52:31.929805 15964 tablet_bootstrap.cc:492] T dfbd1032c876475e974ae5b99a5ce28e P c35e5f91cdd44178b1a78e6c9838563e: No bootstrap required, opened a new log
I20240502 01:52:31.929822 15963 tablet_bootstrap.cc:492] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11: No bootstrap required, opened a new log
I20240502 01:52:31.930029 15965 ts_tablet_manager.cc:1367] T dfbd1032c876475e974ae5b99a5ce28e P 29fba4a87cd543ff99c483f497344b98: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:52:31.930061 15964 ts_tablet_manager.cc:1367] T dfbd1032c876475e974ae5b99a5ce28e P c35e5f91cdd44178b1a78e6c9838563e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240502 01:52:31.930060 15963 ts_tablet_manager.cc:1367] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240502 01:52:31.937510 15965 raft_consensus.cc:348] T dfbd1032c876475e974ae5b99a5ce28e P 29fba4a87cd543ff99c483f497344b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35e5f91cdd44178b1a78e6c9838563e" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 41377 } } peers { permanent_uuid: "135ee4b62c6a443cbbc9e233be847b11" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43863 } } peers { permanent_uuid: "29fba4a87cd543ff99c483f497344b98" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42229 } }
I20240502 01:52:31.937929 15965 raft_consensus.cc:374] T dfbd1032c876475e974ae5b99a5ce28e P 29fba4a87cd543ff99c483f497344b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:31.938031 15965 raft_consensus.cc:729] T dfbd1032c876475e974ae5b99a5ce28e P 29fba4a87cd543ff99c483f497344b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29fba4a87cd543ff99c483f497344b98, State: Initialized, Role: FOLLOWER
I20240502 01:52:31.938778 15965 consensus_queue.cc:260] T dfbd1032c876475e974ae5b99a5ce28e P 29fba4a87cd543ff99c483f497344b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35e5f91cdd44178b1a78e6c9838563e" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 41377 } } peers { permanent_uuid: "135ee4b62c6a443cbbc9e233be847b11" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43863 } } peers { permanent_uuid: "29fba4a87cd543ff99c483f497344b98" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42229 } }
I20240502 01:52:31.939088 15964 raft_consensus.cc:348] T dfbd1032c876475e974ae5b99a5ce28e P c35e5f91cdd44178b1a78e6c9838563e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35e5f91cdd44178b1a78e6c9838563e" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 41377 } } peers { permanent_uuid: "135ee4b62c6a443cbbc9e233be847b11" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43863 } } peers { permanent_uuid: "29fba4a87cd543ff99c483f497344b98" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42229 } }
I20240502 01:52:31.939090 15963 raft_consensus.cc:348] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35e5f91cdd44178b1a78e6c9838563e" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 41377 } } peers { permanent_uuid: "135ee4b62c6a443cbbc9e233be847b11" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43863 } } peers { permanent_uuid: "29fba4a87cd543ff99c483f497344b98" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42229 } }
I20240502 01:52:31.939528 15963 raft_consensus.cc:374] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:31.939528 15964 raft_consensus.cc:374] T dfbd1032c876475e974ae5b99a5ce28e P c35e5f91cdd44178b1a78e6c9838563e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:31.939633 15964 raft_consensus.cc:729] T dfbd1032c876475e974ae5b99a5ce28e P c35e5f91cdd44178b1a78e6c9838563e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35e5f91cdd44178b1a78e6c9838563e, State: Initialized, Role: FOLLOWER
I20240502 01:52:31.939772 15963 raft_consensus.cc:729] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135ee4b62c6a443cbbc9e233be847b11, State: Initialized, Role: FOLLOWER
I20240502 01:52:31.940304 15964 consensus_queue.cc:260] T dfbd1032c876475e974ae5b99a5ce28e P c35e5f91cdd44178b1a78e6c9838563e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35e5f91cdd44178b1a78e6c9838563e" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 41377 } } peers { permanent_uuid: "135ee4b62c6a443cbbc9e233be847b11" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43863 } } peers { permanent_uuid: "29fba4a87cd543ff99c483f497344b98" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42229 } }
I20240502 01:52:31.940372 15963 consensus_queue.cc:260] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35e5f91cdd44178b1a78e6c9838563e" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 41377 } } peers { permanent_uuid: "135ee4b62c6a443cbbc9e233be847b11" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43863 } } peers { permanent_uuid: "29fba4a87cd543ff99c483f497344b98" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42229 } }
I20240502 01:52:31.940603 15965 ts_tablet_manager.cc:1398] T dfbd1032c876475e974ae5b99a5ce28e P 29fba4a87cd543ff99c483f497344b98: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20240502 01:52:31.942862 15964 ts_tablet_manager.cc:1398] T dfbd1032c876475e974ae5b99a5ce28e P c35e5f91cdd44178b1a78e6c9838563e: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20240502 01:52:31.944051 15814 heartbeater.cc:503] Master 127.14.212.254:33883 was elected leader, sending a full tablet report...
I20240502 01:52:31.944478 15963 ts_tablet_manager.cc:1398] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20240502 01:52:32.091645 15815 tablet.cc:2369] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:52:32.104197 15683 tablet.cc:2369] T dfbd1032c876475e974ae5b99a5ce28e P 29fba4a87cd543ff99c483f497344b98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:52:32.131129 15551 tablet.cc:2369] T dfbd1032c876475e974ae5b99a5ce28e P c35e5f91cdd44178b1a78e6c9838563e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:32.159027 15971 raft_consensus.cc:482] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:32.159296 15971 raft_consensus.cc:504] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35e5f91cdd44178b1a78e6c9838563e" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 41377 } } peers { permanent_uuid: "135ee4b62c6a443cbbc9e233be847b11" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43863 } } peers { permanent_uuid: "29fba4a87cd543ff99c483f497344b98" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42229 } }
I20240502 01:52:32.161124 15971 leader_election.cc:288] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c35e5f91cdd44178b1a78e6c9838563e (127.14.212.194:41377), 29fba4a87cd543ff99c483f497344b98 (127.14.212.195:42229)
I20240502 01:52:32.170395 15505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbd1032c876475e974ae5b99a5ce28e" candidate_uuid: "135ee4b62c6a443cbbc9e233be847b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c35e5f91cdd44178b1a78e6c9838563e" is_pre_election: true
I20240502 01:52:32.170395 15636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbd1032c876475e974ae5b99a5ce28e" candidate_uuid: "135ee4b62c6a443cbbc9e233be847b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29fba4a87cd543ff99c483f497344b98" is_pre_election: true
I20240502 01:52:32.171036 15636 raft_consensus.cc:2454] T dfbd1032c876475e974ae5b99a5ce28e P 29fba4a87cd543ff99c483f497344b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 135ee4b62c6a443cbbc9e233be847b11 in term 0.
I20240502 01:52:32.171029 15505 raft_consensus.cc:2454] T dfbd1032c876475e974ae5b99a5ce28e P c35e5f91cdd44178b1a78e6c9838563e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 135ee4b62c6a443cbbc9e233be847b11 in term 0.
I20240502 01:52:32.171741 15702 leader_election.cc:302] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 135ee4b62c6a443cbbc9e233be847b11, c35e5f91cdd44178b1a78e6c9838563e; no voters:
I20240502 01:52:32.172142 15971 raft_consensus.cc:2781] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:32.172341 15971 raft_consensus.cc:482] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:32.172459 15971 raft_consensus.cc:3037] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:32.203219 15971 raft_consensus.cc:504] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35e5f91cdd44178b1a78e6c9838563e" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 41377 } } peers { permanent_uuid: "135ee4b62c6a443cbbc9e233be847b11" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43863 } } peers { permanent_uuid: "29fba4a87cd543ff99c483f497344b98" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42229 } }
I20240502 01:52:32.204088 15971 leader_election.cc:288] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 [CANDIDATE]: Term 1 election: Requested vote from peers c35e5f91cdd44178b1a78e6c9838563e (127.14.212.194:41377), 29fba4a87cd543ff99c483f497344b98 (127.14.212.195:42229)
I20240502 01:52:32.204664 15505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbd1032c876475e974ae5b99a5ce28e" candidate_uuid: "135ee4b62c6a443cbbc9e233be847b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c35e5f91cdd44178b1a78e6c9838563e"
I20240502 01:52:32.204792 15636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbd1032c876475e974ae5b99a5ce28e" candidate_uuid: "135ee4b62c6a443cbbc9e233be847b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29fba4a87cd543ff99c483f497344b98"
I20240502 01:52:32.204954 15505 raft_consensus.cc:3037] T dfbd1032c876475e974ae5b99a5ce28e P c35e5f91cdd44178b1a78e6c9838563e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:32.205045 15636 raft_consensus.cc:3037] T dfbd1032c876475e974ae5b99a5ce28e P 29fba4a87cd543ff99c483f497344b98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:32.303627 15636 raft_consensus.cc:2454] T dfbd1032c876475e974ae5b99a5ce28e P 29fba4a87cd543ff99c483f497344b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 135ee4b62c6a443cbbc9e233be847b11 in term 1.
I20240502 01:52:32.303607 15505 raft_consensus.cc:2454] T dfbd1032c876475e974ae5b99a5ce28e P c35e5f91cdd44178b1a78e6c9838563e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 135ee4b62c6a443cbbc9e233be847b11 in term 1.
I20240502 01:52:32.304307 15700 leader_election.cc:302] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 135ee4b62c6a443cbbc9e233be847b11, 29fba4a87cd543ff99c483f497344b98; no voters:
I20240502 01:52:32.304713 15971 raft_consensus.cc:2781] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:32.305287 15971 raft_consensus.cc:686] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 [term 1 LEADER]: Becoming Leader. State: Replica: 135ee4b62c6a443cbbc9e233be847b11, State: Running, Role: LEADER
I20240502 01:52:32.305927 15971 consensus_queue.cc:237] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35e5f91cdd44178b1a78e6c9838563e" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 41377 } } peers { permanent_uuid: "135ee4b62c6a443cbbc9e233be847b11" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43863 } } peers { permanent_uuid: "29fba4a87cd543ff99c483f497344b98" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42229 } }
I20240502 01:52:32.312752 15228 catalog_manager.cc:5484] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 135ee4b62c6a443cbbc9e233be847b11 (127.14.212.196). New cstate: current_term: 1 leader_uuid: "135ee4b62c6a443cbbc9e233be847b11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35e5f91cdd44178b1a78e6c9838563e" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 41377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "135ee4b62c6a443cbbc9e233be847b11" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29fba4a87cd543ff99c483f497344b98" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42229 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:32.349062 15187 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:52:32.351892 15187 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d7dad9f368e747e192ca2e21396cb1da to finish bootstrapping
I20240502 01:52:32.362105 15187 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c35e5f91cdd44178b1a78e6c9838563e to finish bootstrapping
I20240502 01:52:32.370301 15187 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 29fba4a87cd543ff99c483f497344b98 to finish bootstrapping
I20240502 01:52:32.378569 15187 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 135ee4b62c6a443cbbc9e233be847b11 to finish bootstrapping
I20240502 01:52:32.387409 15187 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e1ef9f8b5e7f4fac914bed0f9a7b9e02 to finish bootstrapping
I20240502 01:52:32.787567 15971 consensus_queue.cc:1035] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29fba4a87cd543ff99c483f497344b98" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:32.806258 15974 consensus_queue.cc:1035] T dfbd1032c876475e974ae5b99a5ce28e P 135ee4b62c6a443cbbc9e233be847b11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c35e5f91cdd44178b1a78e6c9838563e" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:32.828016 15945 heartbeater.cc:503] Master 127.14.212.254:33883 was elected leader, sending a full tablet report...
I20240502 01:52:32.986487 15485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:32.988668 15616 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:32.990681 15748 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:32.991185 15352 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:32.998860 15879 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:33.179229 15616 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:33.186586 15748 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:33.191633 15485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:33.198395 15352 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:33.215144 15879 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:33.358841 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 15289
I20240502 01:52:33.360298 15411 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:33.448104 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 15289
I20240502 01:52:33.471704 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 15421
I20240502 01:52:33.472852 15545 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:33.565243 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 15421
I20240502 01:52:33.583714 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 15553
I20240502 01:52:33.585024 15677 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:33.678074 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 15553
I20240502 01:52:33.695654 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 15685
I20240502 01:52:33.696750 15808 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:33.791882 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 15685
I20240502 01:52:33.810671 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 15817
I20240502 01:52:33.811779 15940 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:33.900864 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 15817
W20240502 01:52:33.914153 15217 connection.cc:535] server connection from 127.14.212.197:60165 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:52:33.914610 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 15199
I20240502 01:52:33.915807 15260 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:34.026189 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 15199
2024-05-02T01:52:34Z chronyd exiting
[ OK ] AdminCliTest.RebalancerReportOnly (5261 ms)
[----------] 1 test from AdminCliTest (5261 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/0
I20240502 01:52:34.071754 15187 test_util.cc:268] Using random seed: -615000928
I20240502 01:52:34.073786 15187 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:52:34.073873 15187 ts_itest-base.cc:113] --------------
I20240502 01:52:34.073930 15187 ts_itest-base.cc:114] 5 tablet servers
I20240502 01:52:34.074002 15187 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:52:34.074062 15187 ts_itest-base.cc:116] --------------
2024-05-02T01:52:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:52:34Z Disabled control of system clock
I20240502 01:52:34.098904 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.254:42513
--webserver_interface=127.14.212.254
--webserver_port=0
--builtin_ntp_servers=127.14.212.212:45865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.254:42513
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 01:52:34.443194 16072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:34.443527 16072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:34.443655 16072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:34.451628 16072 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:52:34.451768 16072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:52:34.451879 16072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:34.451980 16072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:52:34.452049 16072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:52:34.462350 16072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:45865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.14.212.254:42513
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.254:42513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.212.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:34.463807 16072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:34.465731 16072 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:34.477573 16079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:34.478186 16078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:34.478870 16081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:34.479627 16072 server_base.cc:1034] running on GCE node
I20240502 01:52:34.480428 16072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:34.481902 16072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:34.483104 16072 hybrid_clock.cc:648] HybridClock initialized: now 1714614754483038 us; error 50 us; skew 500 ppm
I20240502 01:52:34.483481 16072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:34.485910 16072 webserver.cc:460] Webserver started at http://127.14.212.254:38699/ using document root <none> and password file <none>
I20240502 01:52:34.486706 16072 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:34.486825 16072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:34.487264 16072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:34.490229 16072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1b78b5eaa5454dad96166f380230124a"
format_stamp: "Formatted at 2024-05-02 01:52:34 on dist-test-slave-mk1r"
I20240502 01:52:34.491034 16072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1b78b5eaa5454dad96166f380230124a"
format_stamp: "Formatted at 2024-05-02 01:52:34 on dist-test-slave-mk1r"
I20240502 01:52:34.496627 16072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240502 01:52:34.500273 16088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:34.501796 16072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:34.501960 16072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1b78b5eaa5454dad96166f380230124a"
format_stamp: "Formatted at 2024-05-02 01:52:34 on dist-test-slave-mk1r"
I20240502 01:52:34.502147 16072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:34.521126 16072 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:34.521981 16072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:34.522248 16072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:34.544621 16072 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.254:42513
I20240502 01:52:34.544642 16139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.254:42513 every 8 connection(s)
I20240502 01:52:34.546779 16072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:52:34.547381 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 16072
I20240502 01:52:34.547668 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:52:34.551491 16140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:34.563755 16140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a: Bootstrap starting.
I20240502 01:52:34.566958 16140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:34.567901 16140 log.cc:824] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:34.570392 16140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a: No bootstrap required, opened a new log
I20240502 01:52:34.576758 16140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b78b5eaa5454dad96166f380230124a" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 42513 } }
I20240502 01:52:34.577195 16140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:34.577315 16140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b78b5eaa5454dad96166f380230124a, State: Initialized, Role: FOLLOWER
I20240502 01:52:34.577934 16140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b78b5eaa5454dad96166f380230124a" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 42513 } }
I20240502 01:52:34.578123 16140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:34.578253 16140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:34.578421 16140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:34.632090 16140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b78b5eaa5454dad96166f380230124a" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 42513 } }
I20240502 01:52:34.632807 16140 leader_election.cc:302] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b78b5eaa5454dad96166f380230124a; no voters:
I20240502 01:52:34.633539 16140 leader_election.cc:288] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:34.633772 16145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:34.635042 16145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a [term 1 LEADER]: Becoming Leader. State: Replica: 1b78b5eaa5454dad96166f380230124a, State: Running, Role: LEADER
I20240502 01:52:34.635679 16145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b78b5eaa5454dad96166f380230124a" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 42513 } }
I20240502 01:52:34.636030 16140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:34.639927 16146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b78b5eaa5454dad96166f380230124a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b78b5eaa5454dad96166f380230124a" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 42513 } } }
I20240502 01:52:34.639787 16147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b78b5eaa5454dad96166f380230124a. Latest consensus state: current_term: 1 leader_uuid: "1b78b5eaa5454dad96166f380230124a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b78b5eaa5454dad96166f380230124a" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 42513 } } }
I20240502 01:52:34.640338 16147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:34.640340 16146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:34.644379 16152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:52:34.650215 16152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:52:34.661037 16152 catalog_manager.cc:1347] Generated new cluster ID: 18fe5d10147649a1b895290c209f3392
I20240502 01:52:34.661170 16152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:52:34.677809 16152 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:52:34.679731 16152 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:52:34.707803 16152 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1b78b5eaa5454dad96166f380230124a: Generated new TSK 0
I20240502 01:52:34.708719 16152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:52:34.718065 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.193:0
--local_ip_for_outbound_sockets=127.14.212.193
--webserver_interface=127.14.212.193
--webserver_port=0
--tserver_master_addrs=127.14.212.254:42513
--builtin_ntp_servers=127.14.212.212:45865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:52:35.072407 16164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:35.072693 16164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:35.072815 16164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:35.080653 16164 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:52:35.080778 16164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:35.080979 16164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.193
I20240502 01:52:35.090445 16164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:45865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.212.193
--webserver_port=0
--tserver_master_addrs=127.14.212.254:42513
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.193
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:35.092095 16164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:35.093950 16164 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:35.106212 16169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:35.106832 16170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:35.107412 16172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:35.108109 16164 server_base.cc:1034] running on GCE node
I20240502 01:52:35.108693 16164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:35.109673 16164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:35.110877 16164 hybrid_clock.cc:648] HybridClock initialized: now 1714614755110836 us; error 34 us; skew 500 ppm
I20240502 01:52:35.111186 16164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:35.117028 16164 webserver.cc:460] Webserver started at http://127.14.212.193:40677/ using document root <none> and password file <none>
I20240502 01:52:35.117749 16164 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:35.117875 16164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:35.118192 16164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:35.120713 16164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "963146324d6f4f0893100fd7a269303e"
format_stamp: "Formatted at 2024-05-02 01:52:35 on dist-test-slave-mk1r"
I20240502 01:52:35.121480 16164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "963146324d6f4f0893100fd7a269303e"
format_stamp: "Formatted at 2024-05-02 01:52:35 on dist-test-slave-mk1r"
I20240502 01:52:35.126367 16164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 01:52:35.129720 16178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:35.131021 16164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:52:35.131171 16164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "963146324d6f4f0893100fd7a269303e"
format_stamp: "Formatted at 2024-05-02 01:52:35 on dist-test-slave-mk1r"
I20240502 01:52:35.131363 16164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:35.149353 16164 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:35.150414 16164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:35.150733 16164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:35.151732 16164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:35.153234 16164 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:35.153338 16164 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:35.153455 16164 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:35.153529 16164 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:35.186738 16164 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.193:34531
I20240502 01:52:35.186790 16291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.193:34531 every 8 connection(s)
I20240502 01:52:35.188117 16164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:52:35.189167 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 16164
I20240502 01:52:35.189378 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:52:35.193501 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.194:0
--local_ip_for_outbound_sockets=127.14.212.194
--webserver_interface=127.14.212.194
--webserver_port=0
--tserver_master_addrs=127.14.212.254:42513
--builtin_ntp_servers=127.14.212.212:45865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:52:35.203348 16292 heartbeater.cc:347] Connected to a master server at 127.14.212.254:42513
I20240502 01:52:35.203665 16292 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:35.204408 16292 heartbeater.cc:511] Master 127.14.212.254:42513 requested a full tablet report, sending...
I20240502 01:52:35.206454 16105 ts_manager.cc:194] Registered new tserver with Master: 963146324d6f4f0893100fd7a269303e (127.14.212.193:34531)
I20240502 01:52:35.208109 16105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.193:59613
W20240502 01:52:35.523047 16296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:35.523362 16296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:35.523491 16296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:35.531190 16296 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:52:35.531327 16296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:35.531491 16296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.194
I20240502 01:52:35.541002 16296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:45865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.212.194
--webserver_port=0
--tserver_master_addrs=127.14.212.254:42513
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.194
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:35.542379 16296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:35.544364 16296 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:35.558781 16302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:35.559396 16304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:35.559952 16301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:35.560715 16296 server_base.cc:1034] running on GCE node
I20240502 01:52:35.561367 16296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:35.562412 16296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:35.563638 16296 hybrid_clock.cc:648] HybridClock initialized: now 1714614755563571 us; error 61 us; skew 500 ppm
I20240502 01:52:35.563987 16296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:35.566347 16296 webserver.cc:460] Webserver started at http://127.14.212.194:45143/ using document root <none> and password file <none>
I20240502 01:52:35.567090 16296 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:35.567209 16296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:35.567536 16296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:35.570166 16296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9185c3d72bf84a62a7862c54093e76f0"
format_stamp: "Formatted at 2024-05-02 01:52:35 on dist-test-slave-mk1r"
I20240502 01:52:35.570914 16296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9185c3d72bf84a62a7862c54093e76f0"
format_stamp: "Formatted at 2024-05-02 01:52:35 on dist-test-slave-mk1r"
I20240502 01:52:35.576515 16296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240502 01:52:35.580039 16310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:35.581442 16296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240502 01:52:35.581604 16296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9185c3d72bf84a62a7862c54093e76f0"
format_stamp: "Formatted at 2024-05-02 01:52:35 on dist-test-slave-mk1r"
I20240502 01:52:35.581817 16296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:35.595211 16296 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:35.596114 16296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:35.596403 16296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:35.597419 16296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:35.599001 16296 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:35.599107 16296 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:35.599226 16296 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:35.599301 16296 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:35.632108 16296 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.194:42599
I20240502 01:52:35.632161 16424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.194:42599 every 8 connection(s)
I20240502 01:52:35.633527 16296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:52:35.643872 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 16296
I20240502 01:52:35.644126 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:52:35.646816 16425 heartbeater.cc:347] Connected to a master server at 127.14.212.254:42513
I20240502 01:52:35.647141 16425 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:35.647511 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.195:0
--local_ip_for_outbound_sockets=127.14.212.195
--webserver_interface=127.14.212.195
--webserver_port=0
--tserver_master_addrs=127.14.212.254:42513
--builtin_ntp_servers=127.14.212.212:45865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:52:35.647958 16425 heartbeater.cc:511] Master 127.14.212.254:42513 requested a full tablet report, sending...
I20240502 01:52:35.649451 16105 ts_manager.cc:194] Registered new tserver with Master: 9185c3d72bf84a62a7862c54093e76f0 (127.14.212.194:42599)
I20240502 01:52:35.650628 16105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.194:59843
W20240502 01:52:35.977566 16428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:35.977866 16428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:35.977988 16428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:35.985613 16428 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:52:35.985755 16428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:35.985927 16428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.195
I20240502 01:52:35.995365 16428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:45865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.212.195
--webserver_port=0
--tserver_master_addrs=127.14.212.254:42513
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.195
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:35.996730 16428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:35.998528 16428 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:36.011724 16433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:36.012310 16436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:36.012987 16434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:36.013824 16428 server_base.cc:1034] running on GCE node
I20240502 01:52:36.014464 16428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:36.015486 16428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:36.016737 16428 hybrid_clock.cc:648] HybridClock initialized: now 1714614756016689 us; error 36 us; skew 500 ppm
I20240502 01:52:36.017141 16428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:36.019574 16428 webserver.cc:460] Webserver started at http://127.14.212.195:42761/ using document root <none> and password file <none>
I20240502 01:52:36.020362 16428 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:36.020500 16428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:36.020844 16428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:36.023758 16428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "987669e33ba84c69a55c0355212ca69a"
format_stamp: "Formatted at 2024-05-02 01:52:36 on dist-test-slave-mk1r"
I20240502 01:52:36.024562 16428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "987669e33ba84c69a55c0355212ca69a"
format_stamp: "Formatted at 2024-05-02 01:52:36 on dist-test-slave-mk1r"
I20240502 01:52:36.030216 16428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240502 01:52:36.033926 16442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:36.035324 16428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:52:36.035522 16428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "987669e33ba84c69a55c0355212ca69a"
format_stamp: "Formatted at 2024-05-02 01:52:36 on dist-test-slave-mk1r"
I20240502 01:52:36.035722 16428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:36.073836 16428 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:36.074750 16428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:36.075022 16428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:36.076028 16428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:36.077600 16428 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:36.077695 16428 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:36.077826 16428 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:36.077898 16428 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:36.110672 16428 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.195:44893
I20240502 01:52:36.110736 16555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.195:44893 every 8 connection(s)
I20240502 01:52:36.112028 16428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:52:36.116609 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 16428
I20240502 01:52:36.116792 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:52:36.120183 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.196:0
--local_ip_for_outbound_sockets=127.14.212.196
--webserver_interface=127.14.212.196
--webserver_port=0
--tserver_master_addrs=127.14.212.254:42513
--builtin_ntp_servers=127.14.212.212:45865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:52:36.125639 16556 heartbeater.cc:347] Connected to a master server at 127.14.212.254:42513
I20240502 01:52:36.125950 16556 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:36.126720 16556 heartbeater.cc:511] Master 127.14.212.254:42513 requested a full tablet report, sending...
I20240502 01:52:36.128166 16105 ts_manager.cc:194] Registered new tserver with Master: 987669e33ba84c69a55c0355212ca69a (127.14.212.195:44893)
I20240502 01:52:36.129369 16105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.195:38319
I20240502 01:52:36.210955 16292 heartbeater.cc:503] Master 127.14.212.254:42513 was elected leader, sending a full tablet report...
W20240502 01:52:36.458202 16559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:36.458503 16559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:36.458632 16559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:36.466154 16559 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:52:36.466284 16559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:36.466418 16559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.196
I20240502 01:52:36.475545 16559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:45865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.212.196
--webserver_port=0
--tserver_master_addrs=127.14.212.254:42513
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.196
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:36.476914 16559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:36.478665 16559 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:36.491012 16564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:36.491776 16565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:36.492414 16567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:36.493223 16559 server_base.cc:1034] running on GCE node
I20240502 01:52:36.493984 16559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:36.494957 16559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:36.496220 16559 hybrid_clock.cc:648] HybridClock initialized: now 1714614756496160 us; error 41 us; skew 500 ppm
I20240502 01:52:36.496562 16559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:36.498932 16559 webserver.cc:460] Webserver started at http://127.14.212.196:40739/ using document root <none> and password file <none>
I20240502 01:52:36.499665 16559 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:36.499781 16559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:36.500093 16559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:36.502761 16559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "43cd3aad45bf4580b1744edc3bad825c"
format_stamp: "Formatted at 2024-05-02 01:52:36 on dist-test-slave-mk1r"
I20240502 01:52:36.503484 16559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "43cd3aad45bf4580b1744edc3bad825c"
format_stamp: "Formatted at 2024-05-02 01:52:36 on dist-test-slave-mk1r"
I20240502 01:52:36.508802 16559 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240502 01:52:36.512209 16573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:36.513679 16559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240502 01:52:36.513873 16559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "43cd3aad45bf4580b1744edc3bad825c"
format_stamp: "Formatted at 2024-05-02 01:52:36 on dist-test-slave-mk1r"
I20240502 01:52:36.514077 16559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:36.529232 16559 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:36.530095 16559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:36.530362 16559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:36.531339 16559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:36.533031 16559 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:36.533146 16559 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:36.533272 16559 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:36.533346 16559 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:36.567113 16559 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.196:35503
I20240502 01:52:36.567157 16686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.196:35503 every 8 connection(s)
I20240502 01:52:36.568460 16559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:52:36.569895 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 16559
I20240502 01:52:36.570101 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:52:36.573340 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.197:0
--local_ip_for_outbound_sockets=127.14.212.197
--webserver_interface=127.14.212.197
--webserver_port=0
--tserver_master_addrs=127.14.212.254:42513
--builtin_ntp_servers=127.14.212.212:45865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:52:36.581701 16687 heartbeater.cc:347] Connected to a master server at 127.14.212.254:42513
I20240502 01:52:36.581996 16687 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:36.582710 16687 heartbeater.cc:511] Master 127.14.212.254:42513 requested a full tablet report, sending...
I20240502 01:52:36.584030 16105 ts_manager.cc:194] Registered new tserver with Master: 43cd3aad45bf4580b1744edc3bad825c (127.14.212.196:35503)
I20240502 01:52:36.584806 16105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.196:60593
I20240502 01:52:36.653337 16425 heartbeater.cc:503] Master 127.14.212.254:42513 was elected leader, sending a full tablet report...
W20240502 01:52:36.909854 16690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:36.910125 16690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:36.910214 16690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:36.917575 16690 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:52:36.917692 16690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:36.917810 16690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.197
I20240502 01:52:36.926788 16690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:45865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.212.197
--webserver_port=0
--tserver_master_addrs=127.14.212.254:42513
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.197
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:36.928081 16690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:36.929847 16690 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:36.942312 16696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:36.943048 16697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:36.943636 16699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:36.944424 16690 server_base.cc:1034] running on GCE node
I20240502 01:52:36.945034 16690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:36.946015 16690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:36.947234 16690 hybrid_clock.cc:648] HybridClock initialized: now 1714614756947196 us; error 31 us; skew 500 ppm
I20240502 01:52:36.947558 16690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:36.949784 16690 webserver.cc:460] Webserver started at http://127.14.212.197:34913/ using document root <none> and password file <none>
I20240502 01:52:36.950484 16690 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:36.950608 16690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:36.950937 16690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:36.953495 16690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "2fe6bff357df4e89bfe21d24d12ad62b"
format_stamp: "Formatted at 2024-05-02 01:52:36 on dist-test-slave-mk1r"
I20240502 01:52:36.954278 16690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "2fe6bff357df4e89bfe21d24d12ad62b"
format_stamp: "Formatted at 2024-05-02 01:52:36 on dist-test-slave-mk1r"
I20240502 01:52:36.959733 16690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240502 01:52:36.963158 16705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:36.964427 16690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 01:52:36.964602 16690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "2fe6bff357df4e89bfe21d24d12ad62b"
format_stamp: "Formatted at 2024-05-02 01:52:36 on dist-test-slave-mk1r"
I20240502 01:52:36.964807 16690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:36.983058 16690 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:36.983904 16690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:36.984175 16690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:36.985189 16690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:36.986763 16690 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:36.986861 16690 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:36.987012 16690 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:36.987095 16690 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:37.019368 16690 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.197:38245
I20240502 01:52:37.019423 16817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.197:38245 every 8 connection(s)
I20240502 01:52:37.020725 16690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:52:37.022583 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 16690
I20240502 01:52:37.022785 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:52:37.033586 16818 heartbeater.cc:347] Connected to a master server at 127.14.212.254:42513
I20240502 01:52:37.033843 16818 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:37.034518 16818 heartbeater.cc:511] Master 127.14.212.254:42513 requested a full tablet report, sending...
I20240502 01:52:37.035753 16105 ts_manager.cc:194] Registered new tserver with Master: 2fe6bff357df4e89bfe21d24d12ad62b (127.14.212.197:38245)
I20240502 01:52:37.036631 16105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.197:48691
I20240502 01:52:37.038626 15187 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:52:37.054193 16105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39302:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 01:52:37.087131 16226 tablet_service.cc:1467] Processing CreateTablet for tablet cd0a7c55506240c181aec0c3b9e4e922 (DEFAULT_TABLE table=TestTable [id=41d8fc1b96b345418c2d1a8fd8c18c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:37.086990 16358 tablet_service.cc:1467] Processing CreateTablet for tablet cd0a7c55506240c181aec0c3b9e4e922 (DEFAULT_TABLE table=TestTable [id=41d8fc1b96b345418c2d1a8fd8c18c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:37.088337 16226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd0a7c55506240c181aec0c3b9e4e922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:37.088464 16358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd0a7c55506240c181aec0c3b9e4e922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:37.088402 16753 tablet_service.cc:1467] Processing CreateTablet for tablet cd0a7c55506240c181aec0c3b9e4e922 (DEFAULT_TABLE table=TestTable [id=41d8fc1b96b345418c2d1a8fd8c18c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:37.089794 16753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd0a7c55506240c181aec0c3b9e4e922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:37.098477 16836 tablet_bootstrap.cc:492] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e: Bootstrap starting.
I20240502 01:52:37.101979 16837 tablet_bootstrap.cc:492] T cd0a7c55506240c181aec0c3b9e4e922 P 9185c3d72bf84a62a7862c54093e76f0: Bootstrap starting.
I20240502 01:52:37.102866 16836 tablet_bootstrap.cc:654] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:37.103631 16838 tablet_bootstrap.cc:492] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b: Bootstrap starting.
I20240502 01:52:37.104619 16836 log.cc:824] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:37.105590 16837 tablet_bootstrap.cc:654] T cd0a7c55506240c181aec0c3b9e4e922 P 9185c3d72bf84a62a7862c54093e76f0: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:37.106665 16838 tablet_bootstrap.cc:654] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:37.106834 16837 log.cc:824] T cd0a7c55506240c181aec0c3b9e4e922 P 9185c3d72bf84a62a7862c54093e76f0: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:37.107823 16838 log.cc:824] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:37.107801 16836 tablet_bootstrap.cc:492] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e: No bootstrap required, opened a new log
I20240502 01:52:37.108633 16836 ts_tablet_manager.cc:1367] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 01:52:37.111066 16838 tablet_bootstrap.cc:492] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b: No bootstrap required, opened a new log
I20240502 01:52:37.111405 16838 ts_tablet_manager.cc:1367] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20240502 01:52:37.119695 16838 raft_consensus.cc:348] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9185c3d72bf84a62a7862c54093e76f0" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 42599 } } peers { permanent_uuid: "2fe6bff357df4e89bfe21d24d12ad62b" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 38245 } } peers { permanent_uuid: "963146324d6f4f0893100fd7a269303e" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 34531 } }
I20240502 01:52:37.120070 16838 raft_consensus.cc:374] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:37.120208 16838 raft_consensus.cc:729] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fe6bff357df4e89bfe21d24d12ad62b, State: Initialized, Role: FOLLOWER
I20240502 01:52:37.120519 16837 tablet_bootstrap.cc:492] T cd0a7c55506240c181aec0c3b9e4e922 P 9185c3d72bf84a62a7862c54093e76f0: No bootstrap required, opened a new log
I20240502 01:52:37.120820 16837 ts_tablet_manager.cc:1367] T cd0a7c55506240c181aec0c3b9e4e922 P 9185c3d72bf84a62a7862c54093e76f0: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.009s
I20240502 01:52:37.120859 16838 consensus_queue.cc:260] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9185c3d72bf84a62a7862c54093e76f0" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 42599 } } peers { permanent_uuid: "2fe6bff357df4e89bfe21d24d12ad62b" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 38245 } } peers { permanent_uuid: "963146324d6f4f0893100fd7a269303e" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 34531 } }
I20240502 01:52:37.120744 16836 raft_consensus.cc:348] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9185c3d72bf84a62a7862c54093e76f0" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 42599 } } peers { permanent_uuid: "2fe6bff357df4e89bfe21d24d12ad62b" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 38245 } } peers { permanent_uuid: "963146324d6f4f0893100fd7a269303e" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 34531 } }
I20240502 01:52:37.121237 16836 raft_consensus.cc:374] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:37.121378 16836 raft_consensus.cc:729] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 963146324d6f4f0893100fd7a269303e, State: Initialized, Role: FOLLOWER
I20240502 01:52:37.121979 16818 heartbeater.cc:503] Master 127.14.212.254:42513 was elected leader, sending a full tablet report...
I20240502 01:52:37.122153 16836 consensus_queue.cc:260] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9185c3d72bf84a62a7862c54093e76f0" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 42599 } } peers { permanent_uuid: "2fe6bff357df4e89bfe21d24d12ad62b" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 38245 } } peers { permanent_uuid: "963146324d6f4f0893100fd7a269303e" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 34531 } }
I20240502 01:52:37.123831 16836 ts_tablet_manager.cc:1398] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20240502 01:52:37.124903 16838 ts_tablet_manager.cc:1398] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b: Time spent starting tablet: real 0.013s user 0.007s sys 0.003s
I20240502 01:52:37.129447 16837 raft_consensus.cc:348] T cd0a7c55506240c181aec0c3b9e4e922 P 9185c3d72bf84a62a7862c54093e76f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9185c3d72bf84a62a7862c54093e76f0" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 42599 } } peers { permanent_uuid: "2fe6bff357df4e89bfe21d24d12ad62b" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 38245 } } peers { permanent_uuid: "963146324d6f4f0893100fd7a269303e" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 34531 } }
I20240502 01:52:37.129788 16837 raft_consensus.cc:374] T cd0a7c55506240c181aec0c3b9e4e922 P 9185c3d72bf84a62a7862c54093e76f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:37.129884 16837 raft_consensus.cc:729] T cd0a7c55506240c181aec0c3b9e4e922 P 9185c3d72bf84a62a7862c54093e76f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9185c3d72bf84a62a7862c54093e76f0, State: Initialized, Role: FOLLOWER
I20240502 01:52:37.130441 16837 consensus_queue.cc:260] T cd0a7c55506240c181aec0c3b9e4e922 P 9185c3d72bf84a62a7862c54093e76f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9185c3d72bf84a62a7862c54093e76f0" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 42599 } } peers { permanent_uuid: "2fe6bff357df4e89bfe21d24d12ad62b" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 38245 } } peers { permanent_uuid: "963146324d6f4f0893100fd7a269303e" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 34531 } }
I20240502 01:52:37.131714 16837 ts_tablet_manager.cc:1398] T cd0a7c55506240c181aec0c3b9e4e922 P 9185c3d72bf84a62a7862c54093e76f0: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20240502 01:52:37.131847 16556 heartbeater.cc:503] Master 127.14.212.254:42513 was elected leader, sending a full tablet report...
W20240502 01:52:37.136003 16426 tablet.cc:2369] T cd0a7c55506240c181aec0c3b9e4e922 P 9185c3d72bf84a62a7862c54093e76f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:52:37.191463 16293 tablet.cc:2369] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:52:37.272495 16819 tablet.cc:2369] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:37.363965 16843 raft_consensus.cc:482] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:37.364341 16843 raft_consensus.cc:504] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9185c3d72bf84a62a7862c54093e76f0" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 42599 } } peers { permanent_uuid: "2fe6bff357df4e89bfe21d24d12ad62b" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 38245 } } peers { permanent_uuid: "963146324d6f4f0893100fd7a269303e" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 34531 } }
I20240502 01:52:37.366163 16843 leader_election.cc:288] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9185c3d72bf84a62a7862c54093e76f0 (127.14.212.194:42599), 2fe6bff357df4e89bfe21d24d12ad62b (127.14.212.197:38245)
I20240502 01:52:37.373435 16378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd0a7c55506240c181aec0c3b9e4e922" candidate_uuid: "963146324d6f4f0893100fd7a269303e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9185c3d72bf84a62a7862c54093e76f0" is_pre_election: true
I20240502 01:52:37.373776 16773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd0a7c55506240c181aec0c3b9e4e922" candidate_uuid: "963146324d6f4f0893100fd7a269303e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fe6bff357df4e89bfe21d24d12ad62b" is_pre_election: true
I20240502 01:52:37.373921 16378 raft_consensus.cc:2454] T cd0a7c55506240c181aec0c3b9e4e922 P 9185c3d72bf84a62a7862c54093e76f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 963146324d6f4f0893100fd7a269303e in term 0.
I20240502 01:52:37.374186 16773 raft_consensus.cc:2454] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 963146324d6f4f0893100fd7a269303e in term 0.
I20240502 01:52:37.374579 16180 leader_election.cc:302] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9185c3d72bf84a62a7862c54093e76f0, 963146324d6f4f0893100fd7a269303e; no voters:
I20240502 01:52:37.374934 16843 raft_consensus.cc:2781] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:37.375093 16843 raft_consensus.cc:482] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:37.375197 16843 raft_consensus.cc:3037] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:37.377315 16843 raft_consensus.cc:504] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9185c3d72bf84a62a7862c54093e76f0" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 42599 } } peers { permanent_uuid: "2fe6bff357df4e89bfe21d24d12ad62b" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 38245 } } peers { permanent_uuid: "963146324d6f4f0893100fd7a269303e" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 34531 } }
I20240502 01:52:37.377844 16843 leader_election.cc:288] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e [CANDIDATE]: Term 1 election: Requested vote from peers 9185c3d72bf84a62a7862c54093e76f0 (127.14.212.194:42599), 2fe6bff357df4e89bfe21d24d12ad62b (127.14.212.197:38245)
I20240502 01:52:37.378258 16378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd0a7c55506240c181aec0c3b9e4e922" candidate_uuid: "963146324d6f4f0893100fd7a269303e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9185c3d72bf84a62a7862c54093e76f0"
I20240502 01:52:37.378381 16773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd0a7c55506240c181aec0c3b9e4e922" candidate_uuid: "963146324d6f4f0893100fd7a269303e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fe6bff357df4e89bfe21d24d12ad62b"
I20240502 01:52:37.378474 16378 raft_consensus.cc:3037] T cd0a7c55506240c181aec0c3b9e4e922 P 9185c3d72bf84a62a7862c54093e76f0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:37.378569 16773 raft_consensus.cc:3037] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:37.381592 16378 raft_consensus.cc:2454] T cd0a7c55506240c181aec0c3b9e4e922 P 9185c3d72bf84a62a7862c54093e76f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 963146324d6f4f0893100fd7a269303e in term 1.
I20240502 01:52:37.381592 16773 raft_consensus.cc:2454] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 963146324d6f4f0893100fd7a269303e in term 1.
I20240502 01:52:37.381961 16180 leader_election.cc:302] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9185c3d72bf84a62a7862c54093e76f0, 963146324d6f4f0893100fd7a269303e; no voters:
I20240502 01:52:37.382222 16843 raft_consensus.cc:2781] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:37.382656 16843 raft_consensus.cc:686] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e [term 1 LEADER]: Becoming Leader. State: Replica: 963146324d6f4f0893100fd7a269303e, State: Running, Role: LEADER
I20240502 01:52:37.383108 16843 consensus_queue.cc:237] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9185c3d72bf84a62a7862c54093e76f0" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 42599 } } peers { permanent_uuid: "2fe6bff357df4e89bfe21d24d12ad62b" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 38245 } } peers { permanent_uuid: "963146324d6f4f0893100fd7a269303e" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 34531 } }
I20240502 01:52:37.387501 16104 catalog_manager.cc:5484] T cd0a7c55506240c181aec0c3b9e4e922 P 963146324d6f4f0893100fd7a269303e reported cstate change: term changed from 0 to 1, leader changed from <none> to 963146324d6f4f0893100fd7a269303e (127.14.212.193). New cstate: current_term: 1 leader_uuid: "963146324d6f4f0893100fd7a269303e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9185c3d72bf84a62a7862c54093e76f0" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 42599 } } peers { permanent_uuid: "2fe6bff357df4e89bfe21d24d12ad62b" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 38245 } } peers { permanent_uuid: "963146324d6f4f0893100fd7a269303e" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 34531 } } }
I20240502 01:52:37.456645 15187 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:52:37.458788 15187 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 963146324d6f4f0893100fd7a269303e to finish bootstrapping
I20240502 01:52:37.467487 15187 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9185c3d72bf84a62a7862c54093e76f0 to finish bootstrapping
I20240502 01:52:37.475241 15187 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 987669e33ba84c69a55c0355212ca69a to finish bootstrapping
I20240502 01:52:37.483731 15187 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 43cd3aad45bf4580b1744edc3bad825c to finish bootstrapping
I20240502 01:52:37.491989 15187 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2fe6bff357df4e89bfe21d24d12ad62b to finish bootstrapping
I20240502 01:52:37.499656 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 16164
I20240502 01:52:37.500655 16287 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:37.588177 16687 heartbeater.cc:503] Master 127.14.212.254:42513 was elected leader, sending a full tablet report...
I20240502 01:52:37.589720 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 16164
I20240502 01:52:38.079820 16753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:38.081849 16622 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:38.083415 16358 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:38.091459 16490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:38.223163 16753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:38.229444 16358 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:38.229691 16622 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:38.238943 16490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:38.397186 16753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:38.398782 16622 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:38.400254 16358 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:38.412398 16490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:39.020778 16622 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:39.022296 16358 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:39.023295 16753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:39.035099 16490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:39.163738 16622 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:39.173017 16358 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:39.173594 16753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:39.180583 16490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:39.350841 16753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:39.356562 16622 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:39.357369 16358 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:39.362092 16490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:39.494992 17040 raft_consensus.cc:482] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:39.495255 17040 raft_consensus.cc:504] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9185c3d72bf84a62a7862c54093e76f0" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 42599 } } peers { permanent_uuid: "2fe6bff357df4e89bfe21d24d12ad62b" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 38245 } } peers { permanent_uuid: "963146324d6f4f0893100fd7a269303e" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 34531 } }
I20240502 01:52:39.497283 17040 leader_election.cc:288] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9185c3d72bf84a62a7862c54093e76f0 (127.14.212.194:42599), 963146324d6f4f0893100fd7a269303e (127.14.212.193:34531)
W20240502 01:52:39.499306 16709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.193:34531: connect: Connection refused (error 111)
W20240502 01:52:39.501217 16709 leader_election.cc:334] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 963146324d6f4f0893100fd7a269303e (127.14.212.193:34531): Network error: Client connection negotiation failed: client connection to 127.14.212.193:34531: connect: Connection refused (error 111)
I20240502 01:52:39.504925 16378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd0a7c55506240c181aec0c3b9e4e922" candidate_uuid: "2fe6bff357df4e89bfe21d24d12ad62b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9185c3d72bf84a62a7862c54093e76f0" is_pre_election: true
I20240502 01:52:39.505168 16378 raft_consensus.cc:2454] T cd0a7c55506240c181aec0c3b9e4e922 P 9185c3d72bf84a62a7862c54093e76f0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fe6bff357df4e89bfe21d24d12ad62b in term 1.
I20240502 01:52:39.505699 16707 leader_election.cc:302] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2fe6bff357df4e89bfe21d24d12ad62b, 9185c3d72bf84a62a7862c54093e76f0; no voters: 963146324d6f4f0893100fd7a269303e
I20240502 01:52:39.506186 17040 raft_consensus.cc:2781] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:52:39.506322 17040 raft_consensus.cc:482] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:39.506428 17040 raft_consensus.cc:3037] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:39.508282 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 16296
I20240502 01:52:39.509763 16419 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:39.509727 17040 raft_consensus.cc:504] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9185c3d72bf84a62a7862c54093e76f0" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 42599 } } peers { permanent_uuid: "2fe6bff357df4e89bfe21d24d12ad62b" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 38245 } } peers { permanent_uuid: "963146324d6f4f0893100fd7a269303e" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 34531 } }
I20240502 01:52:39.510421 17040 leader_election.cc:288] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [CANDIDATE]: Term 2 election: Requested vote from peers 9185c3d72bf84a62a7862c54093e76f0 (127.14.212.194:42599), 963146324d6f4f0893100fd7a269303e (127.14.212.193:34531)
W20240502 01:52:39.511831 16709 leader_election.cc:334] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 963146324d6f4f0893100fd7a269303e (127.14.212.193:34531): Network error: Client connection negotiation failed: client connection to 127.14.212.193:34531: connect: Connection refused (error 111)
I20240502 01:52:39.609289 16378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd0a7c55506240c181aec0c3b9e4e922" candidate_uuid: "2fe6bff357df4e89bfe21d24d12ad62b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9185c3d72bf84a62a7862c54093e76f0"
I20240502 01:52:39.609591 16378 raft_consensus.cc:3037] T cd0a7c55506240c181aec0c3b9e4e922 P 9185c3d72bf84a62a7862c54093e76f0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:39.609658 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 16296
I20240502 01:52:39.628208 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 16428
W20240502 01:52:39.629015 16707 leader_election.cc:334] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9185c3d72bf84a62a7862c54093e76f0 (127.14.212.194:42599): Network error: Client connection negotiation failed: client connection to 127.14.212.194:42599: connect: Connection refused (error 111)
I20240502 01:52:39.629180 16707 leader_election.cc:302] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fe6bff357df4e89bfe21d24d12ad62b; no voters: 9185c3d72bf84a62a7862c54093e76f0, 963146324d6f4f0893100fd7a269303e
I20240502 01:52:39.629302 16551 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:39.629479 17040 raft_consensus.cc:2726] T cd0a7c55506240c181aec0c3b9e4e922 P 2fe6bff357df4e89bfe21d24d12ad62b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 01:52:39.723982 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 16428
I20240502 01:52:39.741915 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 16559
I20240502 01:52:39.743043 16682 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:39.836463 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 16559
I20240502 01:52:39.854972 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 16690
I20240502 01:52:39.856067 16812 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:39.955489 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 16690
I20240502 01:52:39.973855 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 16072
I20240502 01:52:39.975085 16135 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:40.095640 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 16072
2024-05-02T01:52:40Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/0 (6070 ms)
[----------] 1 test from RebalanceStartCriteriaTest (6070 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/0
I20240502 01:52:40.141783 15187 test_util.cc:268] Using random seed: -608930880
I20240502 01:52:40.143800 15187 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:52:40.143877 15187 ts_itest-base.cc:113] --------------
I20240502 01:52:40.143945 15187 ts_itest-base.cc:114] 7 tablet servers
I20240502 01:52:40.143996 15187 ts_itest-base.cc:115] 1 replicas per TS
I20240502 01:52:40.144078 15187 ts_itest-base.cc:116] --------------
2024-05-02T01:52:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:52:40Z Disabled control of system clock
I20240502 01:52:40.171012 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.254:43511
--webserver_interface=127.14.212.254
--webserver_port=0
--builtin_ntp_servers=127.14.212.212:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.254:43511
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 01:52:40.506206 17056 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 01:52:40.506533 17056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:40.506631 17056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:40.506747 17056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:40.514241 17056 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:52:40.514358 17056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:52:40.514437 17056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:40.514508 17056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:52:40.514604 17056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:52:40.523736 17056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.14.212.254:43511
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.254:43511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.212.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:40.525148 17056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:40.526903 17056 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:40.538051 17064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:40.538739 17061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:40.539325 17062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:40.540091 17056 server_base.cc:1034] running on GCE node
I20240502 01:52:40.540827 17056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:40.542248 17056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:40.543469 17056 hybrid_clock.cc:648] HybridClock initialized: now 1714614760543393 us; error 54 us; skew 500 ppm
I20240502 01:52:40.543799 17056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:40.551790 17056 webserver.cc:460] Webserver started at http://127.14.212.254:42699/ using document root <none> and password file <none>
I20240502 01:52:40.552784 17056 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:40.552969 17056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:40.553328 17056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:40.556031 17056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0939f6abb7634ecd9c58879083ac7987"
format_stamp: "Formatted at 2024-05-02 01:52:40 on dist-test-slave-mk1r"
I20240502 01:52:40.556778 17056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0939f6abb7634ecd9c58879083ac7987"
format_stamp: "Formatted at 2024-05-02 01:52:40 on dist-test-slave-mk1r"
I20240502 01:52:40.562153 17056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240502 01:52:40.565629 17070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:40.567044 17056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:52:40.567242 17056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0939f6abb7634ecd9c58879083ac7987"
format_stamp: "Formatted at 2024-05-02 01:52:40 on dist-test-slave-mk1r"
I20240502 01:52:40.567426 17056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:40.592386 17056 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:40.593516 17056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:40.593930 17056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:40.613111 17056 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.254:43511
I20240502 01:52:40.613144 17121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.254:43511 every 8 connection(s)
I20240502 01:52:40.614513 17056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:52:40.618377 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 17056
I20240502 01:52:40.618522 17122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.618597 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:52:40.632047 17122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987: Bootstrap starting.
I20240502 01:52:40.635356 17122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.636401 17122 log.cc:824] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:40.638969 17122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987: No bootstrap required, opened a new log
I20240502 01:52:40.645465 17122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0939f6abb7634ecd9c58879083ac7987" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 43511 } }
I20240502 01:52:40.645838 17122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.645925 17122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0939f6abb7634ecd9c58879083ac7987, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.646471 17122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0939f6abb7634ecd9c58879083ac7987" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 43511 } }
I20240502 01:52:40.646678 17122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:40.646803 17122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:40.646970 17122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:40.653508 17122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0939f6abb7634ecd9c58879083ac7987" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 43511 } }
I20240502 01:52:40.654150 17122 leader_election.cc:302] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0939f6abb7634ecd9c58879083ac7987; no voters:
I20240502 01:52:40.654937 17122 leader_election.cc:288] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:40.655090 17127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:40.656567 17127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987 [term 1 LEADER]: Becoming Leader. State: Replica: 0939f6abb7634ecd9c58879083ac7987, State: Running, Role: LEADER
I20240502 01:52:40.657358 17122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:40.657254 17127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0939f6abb7634ecd9c58879083ac7987" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 43511 } }
I20240502 01:52:40.660179 17128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0939f6abb7634ecd9c58879083ac7987" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0939f6abb7634ecd9c58879083ac7987" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 43511 } } }
I20240502 01:52:40.660476 17128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:40.662708 17129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0939f6abb7634ecd9c58879083ac7987. Latest consensus state: current_term: 1 leader_uuid: "0939f6abb7634ecd9c58879083ac7987" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0939f6abb7634ecd9c58879083ac7987" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 43511 } } }
I20240502 01:52:40.663040 17129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:40.664033 17134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:52:40.669792 17134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:52:40.670992 17144 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20240502 01:52:40.678649 17134 catalog_manager.cc:1347] Generated new cluster ID: 41db44e4860744eb83b64e7713bf4d33
I20240502 01:52:40.678772 17134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:52:40.690497 17134 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:52:40.692317 17134 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:52:40.705924 17134 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0939f6abb7634ecd9c58879083ac7987: Generated new TSK 0
I20240502 01:52:40.706777 17134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:52:40.713284 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.193:0
--local_ip_for_outbound_sockets=127.14.212.193
--webserver_interface=127.14.212.193
--webserver_port=0
--tserver_master_addrs=127.14.212.254:43511
--builtin_ntp_servers=127.14.212.212:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:52:41.074342 17146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:41.074697 17146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:41.074831 17146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:41.082475 17146 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:52:41.082607 17146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:41.082772 17146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.193
I20240502 01:52:41.091871 17146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.212.193
--webserver_port=0
--tserver_master_addrs=127.14.212.254:43511
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.193
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:41.093303 17146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:41.095170 17146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:41.108240 17154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:41.108965 17152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:41.109596 17151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:41.110414 17146 server_base.cc:1034] running on GCE node
I20240502 01:52:41.110982 17146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:41.111893 17146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:41.113085 17146 hybrid_clock.cc:648] HybridClock initialized: now 1714614761113036 us; error 45 us; skew 500 ppm
I20240502 01:52:41.113391 17146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:41.115566 17146 webserver.cc:460] Webserver started at http://127.14.212.193:44467/ using document root <none> and password file <none>
I20240502 01:52:41.116366 17146 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:41.116475 17146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:41.116811 17146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:41.119423 17146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38"
format_stamp: "Formatted at 2024-05-02 01:52:41 on dist-test-slave-mk1r"
I20240502 01:52:41.120169 17146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38"
format_stamp: "Formatted at 2024-05-02 01:52:41 on dist-test-slave-mk1r"
I20240502 01:52:41.125468 17146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 01:52:41.129004 17160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:41.130350 17146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:52:41.130491 17146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38"
format_stamp: "Formatted at 2024-05-02 01:52:41 on dist-test-slave-mk1r"
I20240502 01:52:41.130685 17146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:41.148923 17146 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:41.149750 17146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:41.150023 17146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:41.150976 17146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:41.152474 17146 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:41.152562 17146 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:41.152648 17146 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:41.152731 17146 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:41.186452 17146 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.193:44487
I20240502 01:52:41.186502 17274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.193:44487 every 8 connection(s)
I20240502 01:52:41.187762 17146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:52:41.196663 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 17146
I20240502 01:52:41.196944 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:52:41.200836 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.194:0
--local_ip_for_outbound_sockets=127.14.212.194
--webserver_interface=127.14.212.194
--webserver_port=0
--tserver_master_addrs=127.14.212.254:43511
--builtin_ntp_servers=127.14.212.212:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:52:41.201817 17275 heartbeater.cc:347] Connected to a master server at 127.14.212.254:43511
I20240502 01:52:41.202123 17275 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:41.202886 17275 heartbeater.cc:511] Master 127.14.212.254:43511 requested a full tablet report, sending...
I20240502 01:52:41.205020 17087 ts_manager.cc:194] Registered new tserver with Master: b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193:44487)
I20240502 01:52:41.206703 17087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.193:41951
W20240502 01:52:41.531518 17279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:41.531823 17279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:41.531949 17279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:41.539690 17279 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:52:41.539815 17279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:41.539937 17279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.194
I20240502 01:52:41.549176 17279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.212.194
--webserver_port=0
--tserver_master_addrs=127.14.212.254:43511
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.194
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:41.550522 17279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:41.552224 17279 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:41.564759 17284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:41.565296 17287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:41.565989 17285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:41.566816 17279 server_base.cc:1034] running on GCE node
I20240502 01:52:41.567420 17279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:41.568396 17279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:41.569643 17279 hybrid_clock.cc:648] HybridClock initialized: now 1714614761569593 us; error 42 us; skew 500 ppm
I20240502 01:52:41.569981 17279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:41.572340 17279 webserver.cc:460] Webserver started at http://127.14.212.194:36845/ using document root <none> and password file <none>
I20240502 01:52:41.573154 17279 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:41.573273 17279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:41.573598 17279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:41.576192 17279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4d190acabedb48b3bbb8cd6b677622da"
format_stamp: "Formatted at 2024-05-02 01:52:41 on dist-test-slave-mk1r"
I20240502 01:52:41.576975 17279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4d190acabedb48b3bbb8cd6b677622da"
format_stamp: "Formatted at 2024-05-02 01:52:41 on dist-test-slave-mk1r"
I20240502 01:52:41.582028 17279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240502 01:52:41.585507 17293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:41.586871 17279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240502 01:52:41.587042 17279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4d190acabedb48b3bbb8cd6b677622da"
format_stamp: "Formatted at 2024-05-02 01:52:41 on dist-test-slave-mk1r"
I20240502 01:52:41.587250 17279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:41.604054 17279 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:41.604961 17279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:41.605249 17279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:41.606211 17279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:41.607697 17279 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:41.607796 17279 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:41.607918 17279 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:41.607991 17279 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:41.639614 17279 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.194:34515
I20240502 01:52:41.639681 17407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.194:34515 every 8 connection(s)
I20240502 01:52:41.641259 17279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:52:41.649981 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 17279
I20240502 01:52:41.650166 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:52:41.653613 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.195:0
--local_ip_for_outbound_sockets=127.14.212.195
--webserver_interface=127.14.212.195
--webserver_port=0
--tserver_master_addrs=127.14.212.254:43511
--builtin_ntp_servers=127.14.212.212:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:52:41.654474 17408 heartbeater.cc:347] Connected to a master server at 127.14.212.254:43511
I20240502 01:52:41.654778 17408 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:41.655519 17408 heartbeater.cc:511] Master 127.14.212.254:43511 requested a full tablet report, sending...
I20240502 01:52:41.657009 17087 ts_manager.cc:194] Registered new tserver with Master: 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194:34515)
I20240502 01:52:41.658150 17087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.194:46897
W20240502 01:52:41.983665 17411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:41.983960 17411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:41.984056 17411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:41.991595 17411 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:52:41.991719 17411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:41.991837 17411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.195
I20240502 01:52:42.000824 17411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.212.195
--webserver_port=0
--tserver_master_addrs=127.14.212.254:43511
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.195
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:42.002218 17411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:42.003913 17411 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:42.016474 17417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:42.017117 17419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:42.017781 17416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:42.018586 17411 server_base.cc:1034] running on GCE node
I20240502 01:52:42.019172 17411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:42.020192 17411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:42.021399 17411 hybrid_clock.cc:648] HybridClock initialized: now 1714614762021346 us; error 40 us; skew 500 ppm
I20240502 01:52:42.021772 17411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:42.024207 17411 webserver.cc:460] Webserver started at http://127.14.212.195:39119/ using document root <none> and password file <none>
I20240502 01:52:42.025066 17411 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:42.025202 17411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:42.025527 17411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:42.028213 17411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75"
format_stamp: "Formatted at 2024-05-02 01:52:42 on dist-test-slave-mk1r"
I20240502 01:52:42.029006 17411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75"
format_stamp: "Formatted at 2024-05-02 01:52:42 on dist-test-slave-mk1r"
I20240502 01:52:42.034085 17411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 01:52:42.037693 17425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:42.039073 17411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:52:42.039242 17411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75"
format_stamp: "Formatted at 2024-05-02 01:52:42 on dist-test-slave-mk1r"
I20240502 01:52:42.039440 17411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:42.075008 17411 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:42.075892 17411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:42.076193 17411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:42.077193 17411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:42.078701 17411 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:42.078807 17411 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:42.078928 17411 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:42.079003 17411 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:42.112601 17411 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.195:42281
I20240502 01:52:42.112746 17538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.195:42281 every 8 connection(s)
I20240502 01:52:42.113965 17411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:52:42.123180 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 17411
I20240502 01:52:42.123376 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:52:42.126730 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.196:0
--local_ip_for_outbound_sockets=127.14.212.196
--webserver_interface=127.14.212.196
--webserver_port=0
--tserver_master_addrs=127.14.212.254:43511
--builtin_ntp_servers=127.14.212.212:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:52:42.127032 17539 heartbeater.cc:347] Connected to a master server at 127.14.212.254:43511
I20240502 01:52:42.127285 17539 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:42.128023 17539 heartbeater.cc:511] Master 127.14.212.254:43511 requested a full tablet report, sending...
I20240502 01:52:42.129469 17087 ts_manager.cc:194] Registered new tserver with Master: d4e4a5c2ba664cd08c3aba1bb348ed75 (127.14.212.195:42281)
I20240502 01:52:42.130273 17087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.195:39947
I20240502 01:52:42.209766 17275 heartbeater.cc:503] Master 127.14.212.254:43511 was elected leader, sending a full tablet report...
W20240502 01:52:42.458614 17542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:42.458956 17542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:42.459111 17542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:42.466881 17542 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:52:42.467036 17542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:42.467202 17542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.196
I20240502 01:52:42.476408 17542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.212.196
--webserver_port=0
--tserver_master_addrs=127.14.212.254:43511
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.196
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:42.477815 17542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:42.479629 17542 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:42.492547 17548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:42.493435 17542 server_base.cc:1034] running on GCE node
W20240502 01:52:42.494067 17550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:42.494647 17547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:42.495368 17542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:42.496353 17542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:42.497653 17542 hybrid_clock.cc:648] HybridClock initialized: now 1714614762497589 us; error 49 us; skew 500 ppm
I20240502 01:52:42.498020 17542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:42.500406 17542 webserver.cc:460] Webserver started at http://127.14.212.196:40389/ using document root <none> and password file <none>
I20240502 01:52:42.501210 17542 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:42.501334 17542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:42.501644 17542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:42.504413 17542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "d21ca31c11b746d99a102d116cfad615"
format_stamp: "Formatted at 2024-05-02 01:52:42 on dist-test-slave-mk1r"
I20240502 01:52:42.505241 17542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "d21ca31c11b746d99a102d116cfad615"
format_stamp: "Formatted at 2024-05-02 01:52:42 on dist-test-slave-mk1r"
I20240502 01:52:42.510396 17542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20240502 01:52:42.513873 17556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:42.515314 17542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:52:42.515484 17542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "d21ca31c11b746d99a102d116cfad615"
format_stamp: "Formatted at 2024-05-02 01:52:42 on dist-test-slave-mk1r"
I20240502 01:52:42.515691 17542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:42.530987 17542 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:42.531805 17542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:42.532078 17542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:42.533113 17542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:42.534626 17542 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:42.534744 17542 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:42.534870 17542 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:42.534955 17542 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:42.567102 17542 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.196:43697
I20240502 01:52:42.567170 17669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.196:43697 every 8 connection(s)
I20240502 01:52:42.568398 17542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:52:42.575670 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 17542
I20240502 01:52:42.575861 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:52:42.578896 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.197:0
--local_ip_for_outbound_sockets=127.14.212.197
--webserver_interface=127.14.212.197
--webserver_port=0
--tserver_master_addrs=127.14.212.254:43511
--builtin_ntp_servers=127.14.212.212:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:52:42.581400 17670 heartbeater.cc:347] Connected to a master server at 127.14.212.254:43511
I20240502 01:52:42.581728 17670 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:42.582460 17670 heartbeater.cc:511] Master 127.14.212.254:43511 requested a full tablet report, sending...
I20240502 01:52:42.583914 17087 ts_manager.cc:194] Registered new tserver with Master: d21ca31c11b746d99a102d116cfad615 (127.14.212.196:43697)
I20240502 01:52:42.585114 17087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.196:47177
I20240502 01:52:42.660562 17408 heartbeater.cc:503] Master 127.14.212.254:43511 was elected leader, sending a full tablet report...
W20240502 01:52:42.916712 17673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:42.917096 17673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:42.917243 17673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:42.924947 17673 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:52:42.925107 17673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:42.925271 17673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.197
I20240502 01:52:42.934317 17673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.212.197
--webserver_port=0
--tserver_master_addrs=127.14.212.254:43511
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.197
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:42.935725 17673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:42.937533 17673 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:42.949792 17679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:42.950448 17678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:42.951064 17681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:42.951833 17673 server_base.cc:1034] running on GCE node
I20240502 01:52:42.952394 17673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:42.953356 17673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:42.954583 17673 hybrid_clock.cc:648] HybridClock initialized: now 1714614762954522 us; error 44 us; skew 500 ppm
I20240502 01:52:42.954912 17673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:42.957161 17673 webserver.cc:460] Webserver started at http://127.14.212.197:36535/ using document root <none> and password file <none>
I20240502 01:52:42.957907 17673 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:42.958024 17673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:42.958336 17673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:42.961032 17673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "adf51a82733d4010b3028cb54cc307e2"
format_stamp: "Formatted at 2024-05-02 01:52:42 on dist-test-slave-mk1r"
I20240502 01:52:42.961799 17673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "adf51a82733d4010b3028cb54cc307e2"
format_stamp: "Formatted at 2024-05-02 01:52:42 on dist-test-slave-mk1r"
I20240502 01:52:42.966995 17673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 01:52:42.970532 17687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:42.971997 17673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 01:52:42.972165 17673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "adf51a82733d4010b3028cb54cc307e2"
format_stamp: "Formatted at 2024-05-02 01:52:42 on dist-test-slave-mk1r"
I20240502 01:52:42.972378 17673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:43.009899 17673 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:43.010919 17673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:43.011206 17673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:43.012214 17673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:43.014037 17673 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:43.014150 17673 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:43.014281 17673 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:43.014355 17673 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:43.047608 17673 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.197:41667
I20240502 01:52:43.047681 17799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.197:41667 every 8 connection(s)
I20240502 01:52:43.048975 17673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:52:43.051394 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 17673
I20240502 01:52:43.051586 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:52:43.055107 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.198:0
--local_ip_for_outbound_sockets=127.14.212.198
--webserver_interface=127.14.212.198
--webserver_port=0
--tserver_master_addrs=127.14.212.254:43511
--builtin_ntp_servers=127.14.212.212:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:52:43.063658 17800 heartbeater.cc:347] Connected to a master server at 127.14.212.254:43511
I20240502 01:52:43.063970 17800 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:43.064699 17800 heartbeater.cc:511] Master 127.14.212.254:43511 requested a full tablet report, sending...
I20240502 01:52:43.066036 17087 ts_manager.cc:194] Registered new tserver with Master: adf51a82733d4010b3028cb54cc307e2 (127.14.212.197:41667)
I20240502 01:52:43.066850 17087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.197:59543
I20240502 01:52:43.132966 17539 heartbeater.cc:503] Master 127.14.212.254:43511 was elected leader, sending a full tablet report...
W20240502 01:52:43.391049 17804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:43.391382 17804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:43.391491 17804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:43.399163 17804 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:52:43.399322 17804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:43.399438 17804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.198
I20240502 01:52:43.408746 17804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.14.212.198
--webserver_port=0
--tserver_master_addrs=127.14.212.254:43511
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.198
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:43.410095 17804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:43.411908 17804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:43.425314 17809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:43.426026 17812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:43.426709 17810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:43.428088 17804 server_base.cc:1034] running on GCE node
I20240502 01:52:43.428814 17804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:43.429857 17804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:43.431090 17804 hybrid_clock.cc:648] HybridClock initialized: now 1714614763431023 us; error 62 us; skew 500 ppm
I20240502 01:52:43.431440 17804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:43.433845 17804 webserver.cc:460] Webserver started at http://127.14.212.198:42501/ using document root <none> and password file <none>
I20240502 01:52:43.434635 17804 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:43.434758 17804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:43.435072 17804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:43.437808 17804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a"
format_stamp: "Formatted at 2024-05-02 01:52:43 on dist-test-slave-mk1r"
I20240502 01:52:43.438608 17804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a"
format_stamp: "Formatted at 2024-05-02 01:52:43 on dist-test-slave-mk1r"
I20240502 01:52:43.444072 17804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240502 01:52:43.447784 17818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:43.449294 17804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:52:43.449471 17804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a"
format_stamp: "Formatted at 2024-05-02 01:52:43 on dist-test-slave-mk1r"
I20240502 01:52:43.449683 17804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:43.480342 17804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:43.481328 17804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:43.481619 17804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:43.482617 17804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:43.484215 17804 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:43.484324 17804 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:43.484452 17804 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:43.484525 17804 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:43.518041 17804 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.198:34487
I20240502 01:52:43.518110 17930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.198:34487 every 8 connection(s)
I20240502 01:52:43.519429 17804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:52:43.525218 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 17804
I20240502 01:52:43.525422 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:52:43.528862 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.199:0
--local_ip_for_outbound_sockets=127.14.212.199
--webserver_interface=127.14.212.199
--webserver_port=0
--tserver_master_addrs=127.14.212.254:43511
--builtin_ntp_servers=127.14.212.212:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:52:43.534489 17931 heartbeater.cc:347] Connected to a master server at 127.14.212.254:43511
I20240502 01:52:43.534848 17931 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:43.535769 17931 heartbeater.cc:511] Master 127.14.212.254:43511 requested a full tablet report, sending...
I20240502 01:52:43.537377 17087 ts_manager.cc:194] Registered new tserver with Master: aaf6d9dc6f5747bd83d2b9e24a7d679a (127.14.212.198:34487)
I20240502 01:52:43.538173 17087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.198:38739
I20240502 01:52:43.588392 17670 heartbeater.cc:503] Master 127.14.212.254:43511 was elected leader, sending a full tablet report...
W20240502 01:52:43.876950 17934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:43.877259 17934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:43.877408 17934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:43.885118 17934 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:52:43.885258 17934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:43.885421 17934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.199
I20240502 01:52:43.895059 17934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.14.212.199
--webserver_port=0
--tserver_master_addrs=127.14.212.254:43511
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.199
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:43.896402 17934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:43.898154 17934 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:43.910676 17940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:43.911235 17942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:43.911903 17939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:43.912658 17934 server_base.cc:1034] running on GCE node
I20240502 01:52:43.913341 17934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:43.914330 17934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:43.915534 17934 hybrid_clock.cc:648] HybridClock initialized: now 1714614763915489 us; error 42 us; skew 500 ppm
I20240502 01:52:43.915859 17934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:43.918160 17934 webserver.cc:460] Webserver started at http://127.14.212.199:41373/ using document root <none> and password file <none>
I20240502 01:52:43.918898 17934 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:43.919013 17934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:43.919338 17934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:43.921988 17934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "f7baee4920524734aa33ce500a559f3b"
format_stamp: "Formatted at 2024-05-02 01:52:43 on dist-test-slave-mk1r"
I20240502 01:52:43.922720 17934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "f7baee4920524734aa33ce500a559f3b"
format_stamp: "Formatted at 2024-05-02 01:52:43 on dist-test-slave-mk1r"
I20240502 01:52:43.927613 17934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 01:52:43.930959 17948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:43.932287 17934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:52:43.932452 17934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f7baee4920524734aa33ce500a559f3b"
format_stamp: "Formatted at 2024-05-02 01:52:43 on dist-test-slave-mk1r"
I20240502 01:52:43.932643 17934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:43.960930 17934 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:43.961804 17934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:43.962072 17934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:43.963088 17934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:43.964689 17934 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:43.964779 17934 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:43.964921 17934 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:43.964996 17934 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:43.997673 17934 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.199:45747
I20240502 01:52:43.997731 18062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.199:45747 every 8 connection(s)
I20240502 01:52:43.999003 17934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:52:44.001116 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 17934
I20240502 01:52:44.001335 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 01:52:44.011761 18063 heartbeater.cc:347] Connected to a master server at 127.14.212.254:43511
I20240502 01:52:44.012059 18063 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:44.012791 18063 heartbeater.cc:511] Master 127.14.212.254:43511 requested a full tablet report, sending...
I20240502 01:52:44.014194 17087 ts_manager.cc:194] Registered new tserver with Master: f7baee4920524734aa33ce500a559f3b (127.14.212.199:45747)
I20240502 01:52:44.014988 17087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.199:49235
I20240502 01:52:44.017685 15187 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 01:52:44.034063 17087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53564:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 01:52:44.065142 17473 tablet_service.cc:1467] Processing CreateTablet for tablet a487f57191fe4eb998ad8f5f86f4050b (DEFAULT_TABLE table=TestTable [id=2b75d4b424644aa4a9217094436a897e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:44.066438 17473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a487f57191fe4eb998ad8f5f86f4050b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:44.069171 17800 heartbeater.cc:503] Master 127.14.212.254:43511 was elected leader, sending a full tablet report...
I20240502 01:52:44.076730 18078 tablet_bootstrap.cc:492] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75: Bootstrap starting.
I20240502 01:52:44.079782 18078 tablet_bootstrap.cc:654] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:44.080819 18078 log.cc:824] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:44.083298 18078 tablet_bootstrap.cc:492] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75: No bootstrap required, opened a new log
I20240502 01:52:44.083554 18078 ts_tablet_manager.cc:1367] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 01:52:44.089344 18078 raft_consensus.cc:348] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } }
I20240502 01:52:44.089669 18078 raft_consensus.cc:374] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:44.089774 18078 raft_consensus.cc:729] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e4a5c2ba664cd08c3aba1bb348ed75, State: Initialized, Role: FOLLOWER
I20240502 01:52:44.090395 18078 consensus_queue.cc:260] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } }
I20240502 01:52:44.090649 18078 raft_consensus.cc:388] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:44.090814 18078 raft_consensus.cc:482] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:44.090986 18078 raft_consensus.cc:3037] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:44.093127 18078 raft_consensus.cc:504] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } }
I20240502 01:52:44.093576 18078 leader_election.cc:302] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4e4a5c2ba664cd08c3aba1bb348ed75; no voters:
I20240502 01:52:44.094154 18078 leader_election.cc:288] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:44.094278 18080 raft_consensus.cc:2781] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:44.094938 18080 raft_consensus.cc:686] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 1 LEADER]: Becoming Leader. State: Replica: d4e4a5c2ba664cd08c3aba1bb348ed75, State: Running, Role: LEADER
I20240502 01:52:44.095491 18080 consensus_queue.cc:237] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } }
I20240502 01:52:44.096010 18078 ts_tablet_manager.cc:1398] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20240502 01:52:44.100793 17087 catalog_manager.cc:5484] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4e4a5c2ba664cd08c3aba1bb348ed75 (127.14.212.195). New cstate: current_term: 1 leader_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } } }
I20240502 01:52:44.119971 15187 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 01:52:44.122134 15187 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b8ffc52e71f045e5ac0f8aa35f3eea38 to finish bootstrapping
I20240502 01:52:44.131635 15187 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4d190acabedb48b3bbb8cd6b677622da to finish bootstrapping
I20240502 01:52:44.141260 15187 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d4e4a5c2ba664cd08c3aba1bb348ed75 to finish bootstrapping
I20240502 01:52:44.149596 15187 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d21ca31c11b746d99a102d116cfad615 to finish bootstrapping
I20240502 01:52:44.158596 15187 ts_itest-base.cc:233] Waiting for 0 tablets on tserver adf51a82733d4010b3028cb54cc307e2 to finish bootstrapping
I20240502 01:52:44.167896 15187 ts_itest-base.cc:233] Waiting for 0 tablets on tserver aaf6d9dc6f5747bd83d2b9e24a7d679a to finish bootstrapping
I20240502 01:52:44.176435 15187 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f7baee4920524734aa33ce500a559f3b to finish bootstrapping
I20240502 01:52:44.185299 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 17411
I20240502 01:52:44.186374 17534 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:44.274993 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 17411
I20240502 01:52:44.292282 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 17542
I20240502 01:52:44.293596 17665 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:44.375083 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 17542
I20240502 01:52:44.391757 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 17673
I20240502 01:52:44.392961 17795 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:44.473819 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 17673
I20240502 01:52:44.490073 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 17804
I20240502 01:52:44.491367 17926 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:44.571594 17931 heartbeater.cc:503] Master 127.14.212.254:43511 was elected leader, sending a full tablet report...
I20240502 01:52:44.572454 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 17804
W20240502 01:52:44.588719 17072 connection.cc:535] server connection from 127.14.212.198:38739 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:52:44.589151 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 17934
I20240502 01:52:44.590447 18058 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:44.687623 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 17934
I20240502 01:52:48.458492 17087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53564:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:52:48.481642 17208 tablet_service.cc:1467] Processing CreateTablet for tablet 1e8092e428084a56bbfaaca963877631 (DEFAULT_TABLE table=rebalance_test_table_0 [id=edade0db079c4e7b9fa4c7a565260d19]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:48.481632 17342 tablet_service.cc:1467] Processing CreateTablet for tablet 4d4eaec689d345b4a812a448e5d359e4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=edade0db079c4e7b9fa4c7a565260d19]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:48.481693 17209 tablet_service.cc:1467] Processing CreateTablet for tablet d94d0492ef1d4603b77441d09ad3c6dc (DEFAULT_TABLE table=rebalance_test_table_0 [id=edade0db079c4e7b9fa4c7a565260d19]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:48.483286 17342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d4eaec689d345b4a812a448e5d359e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:48.483282 17208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e8092e428084a56bbfaaca963877631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:48.483878 17209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94d0492ef1d4603b77441d09ad3c6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:48.497006 18106 tablet_bootstrap.cc:492] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da: Bootstrap starting.
I20240502 01:52:48.497155 18105 tablet_bootstrap.cc:492] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38: Bootstrap starting.
I20240502 01:52:48.499961 18105 tablet_bootstrap.cc:654] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:48.499969 18106 tablet_bootstrap.cc:654] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:48.501120 18106 log.cc:824] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:48.501123 18105 log.cc:824] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:48.503916 18105 tablet_bootstrap.cc:492] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38: No bootstrap required, opened a new log
I20240502 01:52:48.503928 18106 tablet_bootstrap.cc:492] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da: No bootstrap required, opened a new log
I20240502 01:52:48.504216 18106 ts_tablet_manager.cc:1367] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240502 01:52:48.504210 18105 ts_tablet_manager.cc:1367] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 01:52:48.510494 18105 raft_consensus.cc:348] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:48.510494 18106 raft_consensus.cc:348] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:48.510875 18106 raft_consensus.cc:374] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:48.510890 18105 raft_consensus.cc:374] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:48.510995 18106 raft_consensus.cc:729] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d190acabedb48b3bbb8cd6b677622da, State: Initialized, Role: FOLLOWER
I20240502 01:52:48.510998 18105 raft_consensus.cc:729] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Initialized, Role: FOLLOWER
I20240502 01:52:48.511670 18105 consensus_queue.cc:260] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:48.511667 18106 consensus_queue.cc:260] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:48.511879 18106 raft_consensus.cc:388] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:48.511883 18105 raft_consensus.cc:388] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:48.512020 18105 raft_consensus.cc:482] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:48.512027 18106 raft_consensus.cc:482] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:48.512180 18106 raft_consensus.cc:3037] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:48.512195 18105 raft_consensus.cc:3037] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:48.514757 18106 raft_consensus.cc:504] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:48.514760 18105 raft_consensus.cc:504] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:48.515340 18105 leader_election.cc:302] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8ffc52e71f045e5ac0f8aa35f3eea38; no voters:
I20240502 01:52:48.515326 18106 leader_election.cc:302] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d190acabedb48b3bbb8cd6b677622da; no voters:
I20240502 01:52:48.516093 18105 leader_election.cc:288] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:48.516110 18106 leader_election.cc:288] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:48.516254 18110 raft_consensus.cc:2781] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:48.516330 18109 raft_consensus.cc:2781] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:48.522257 18109 raft_consensus.cc:686] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Becoming Leader. State: Replica: 4d190acabedb48b3bbb8cd6b677622da, State: Running, Role: LEADER
I20240502 01:52:48.523204 18109 consensus_queue.cc:237] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:48.525142 18106 ts_tablet_manager.cc:1398] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da: Time spent starting tablet: real 0.021s user 0.017s sys 0.005s
I20240502 01:52:48.525781 18105 ts_tablet_manager.cc:1398] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38: Time spent starting tablet: real 0.021s user 0.010s sys 0.010s
I20240502 01:52:48.525923 18110 raft_consensus.cc:686] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Becoming Leader. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Running, Role: LEADER
I20240502 01:52:48.526568 18110 consensus_queue.cc:237] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:48.529572 18105 tablet_bootstrap.cc:492] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38: Bootstrap starting.
I20240502 01:52:48.529547 17087 catalog_manager.cc:5484] T 4d4eaec689d345b4a812a448e5d359e4 P 4d190acabedb48b3bbb8cd6b677622da reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194). New cstate: current_term: 1 leader_uuid: "4d190acabedb48b3bbb8cd6b677622da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } }
I20240502 01:52:48.531738 18105 tablet_bootstrap.cc:654] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:48.531637 17086 catalog_manager.cc:5484] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } }
I20240502 01:52:48.534303 18105 tablet_bootstrap.cc:492] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38: No bootstrap required, opened a new log
I20240502 01:52:48.534487 18105 ts_tablet_manager.cc:1367] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 01:52:48.535740 18105 raft_consensus.cc:348] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:48.535949 18105 raft_consensus.cc:374] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:48.536085 18105 raft_consensus.cc:729] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Initialized, Role: FOLLOWER
I20240502 01:52:48.536412 18105 consensus_queue.cc:260] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:48.536617 18105 raft_consensus.cc:388] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:48.536711 18105 raft_consensus.cc:482] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:48.536828 18105 raft_consensus.cc:3037] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:48.539448 18105 raft_consensus.cc:504] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:48.539709 18105 leader_election.cc:302] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8ffc52e71f045e5ac0f8aa35f3eea38; no voters:
I20240502 01:52:48.539894 18105 leader_election.cc:288] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:48.540023 18111 raft_consensus.cc:2781] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:48.540199 18111 raft_consensus.cc:686] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Becoming Leader. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Running, Role: LEADER
I20240502 01:52:48.540383 18105 ts_tablet_manager.cc:1398] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240502 01:52:48.540537 18111 consensus_queue.cc:237] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:48.542757 17086 catalog_manager.cc:5484] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } }
W20240502 01:52:48.902148 18122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:48.902501 18122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:48.910156 18122 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 135.791 ms
time per row: 0.0678955 ms
I20240502 01:52:49.196305 17084 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53564:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:52:49.205474 17342 tablet_service.cc:1467] Processing CreateTablet for tablet 2bfc0209bd434fa9a56b45d7a9b73a12 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3f2e7f170c6a46d283b6e6cd5b54201e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:49.205945 17342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bfc0209bd434fa9a56b45d7a9b73a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:49.206079 17341 tablet_service.cc:1467] Processing CreateTablet for tablet b477b96739354b6ab43bf4ee230f4559 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3f2e7f170c6a46d283b6e6cd5b54201e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:49.206588 17341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b477b96739354b6ab43bf4ee230f4559. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:49.208758 17208 tablet_service.cc:1467] Processing CreateTablet for tablet c130cb706b004c2088925fba192a8c90 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3f2e7f170c6a46d283b6e6cd5b54201e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:49.209223 17208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c130cb706b004c2088925fba192a8c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:49.212549 18169 tablet_bootstrap.cc:492] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da: Bootstrap starting.
I20240502 01:52:49.214267 18169 tablet_bootstrap.cc:654] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:49.215286 18170 tablet_bootstrap.cc:492] T c130cb706b004c2088925fba192a8c90 P b8ffc52e71f045e5ac0f8aa35f3eea38: Bootstrap starting.
I20240502 01:52:49.216148 18169 tablet_bootstrap.cc:492] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da: No bootstrap required, opened a new log
I20240502 01:52:49.216284 18169 ts_tablet_manager.cc:1367] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:52:49.216797 18170 tablet_bootstrap.cc:654] T c130cb706b004c2088925fba192a8c90 P b8ffc52e71f045e5ac0f8aa35f3eea38: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:49.217316 18169 raft_consensus.cc:348] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:49.217509 18169 raft_consensus.cc:374] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:49.217605 18169 raft_consensus.cc:729] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d190acabedb48b3bbb8cd6b677622da, State: Initialized, Role: FOLLOWER
I20240502 01:52:49.217908 18169 consensus_queue.cc:260] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:49.218073 18169 raft_consensus.cc:388] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:49.218164 18169 raft_consensus.cc:482] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:49.218278 18169 raft_consensus.cc:3037] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:49.218750 18170 tablet_bootstrap.cc:492] T c130cb706b004c2088925fba192a8c90 P b8ffc52e71f045e5ac0f8aa35f3eea38: No bootstrap required, opened a new log
I20240502 01:52:49.218873 18170 ts_tablet_manager.cc:1367] T c130cb706b004c2088925fba192a8c90 P b8ffc52e71f045e5ac0f8aa35f3eea38: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:52:49.219471 18170 raft_consensus.cc:348] T c130cb706b004c2088925fba192a8c90 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:49.219609 18170 raft_consensus.cc:374] T c130cb706b004c2088925fba192a8c90 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:49.219694 18170 raft_consensus.cc:729] T c130cb706b004c2088925fba192a8c90 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Initialized, Role: FOLLOWER
I20240502 01:52:49.219909 18170 consensus_queue.cc:260] T c130cb706b004c2088925fba192a8c90 P b8ffc52e71f045e5ac0f8aa35f3eea38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:49.220063 18170 raft_consensus.cc:388] T c130cb706b004c2088925fba192a8c90 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:49.220150 18170 raft_consensus.cc:482] T c130cb706b004c2088925fba192a8c90 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:49.220275 18170 raft_consensus.cc:3037] T c130cb706b004c2088925fba192a8c90 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:49.220679 18169 raft_consensus.cc:504] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:49.220947 18169 leader_election.cc:302] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d190acabedb48b3bbb8cd6b677622da; no voters:
I20240502 01:52:49.221155 18169 leader_election.cc:288] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:49.221472 18159 raft_consensus.cc:2781] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:49.222059 18169 ts_tablet_manager.cc:1398] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 01:52:49.222321 18169 tablet_bootstrap.cc:492] T 2bfc0209bd434fa9a56b45d7a9b73a12 P 4d190acabedb48b3bbb8cd6b677622da: Bootstrap starting.
I20240502 01:52:49.223373 18170 raft_consensus.cc:504] T c130cb706b004c2088925fba192a8c90 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:49.223713 18169 tablet_bootstrap.cc:654] T 2bfc0209bd434fa9a56b45d7a9b73a12 P 4d190acabedb48b3bbb8cd6b677622da: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:49.223704 18170 leader_election.cc:302] T c130cb706b004c2088925fba192a8c90 P b8ffc52e71f045e5ac0f8aa35f3eea38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8ffc52e71f045e5ac0f8aa35f3eea38; no voters:
I20240502 01:52:49.223946 18170 leader_election.cc:288] T c130cb706b004c2088925fba192a8c90 P b8ffc52e71f045e5ac0f8aa35f3eea38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:49.224066 18155 raft_consensus.cc:2781] T c130cb706b004c2088925fba192a8c90 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:49.225047 18170 ts_tablet_manager.cc:1398] T c130cb706b004c2088925fba192a8c90 P b8ffc52e71f045e5ac0f8aa35f3eea38: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 01:52:49.225795 18159 raft_consensus.cc:686] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Becoming Leader. State: Replica: 4d190acabedb48b3bbb8cd6b677622da, State: Running, Role: LEADER
I20240502 01:52:49.226135 18159 consensus_queue.cc:237] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:49.226783 18169 tablet_bootstrap.cc:492] T 2bfc0209bd434fa9a56b45d7a9b73a12 P 4d190acabedb48b3bbb8cd6b677622da: No bootstrap required, opened a new log
I20240502 01:52:49.226970 18169 ts_tablet_manager.cc:1367] T 2bfc0209bd434fa9a56b45d7a9b73a12 P 4d190acabedb48b3bbb8cd6b677622da: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 01:52:49.227823 18169 raft_consensus.cc:348] T 2bfc0209bd434fa9a56b45d7a9b73a12 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:49.228045 18169 raft_consensus.cc:374] T 2bfc0209bd434fa9a56b45d7a9b73a12 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:49.228158 18169 raft_consensus.cc:729] T 2bfc0209bd434fa9a56b45d7a9b73a12 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d190acabedb48b3bbb8cd6b677622da, State: Initialized, Role: FOLLOWER
I20240502 01:52:49.228464 18169 consensus_queue.cc:260] T 2bfc0209bd434fa9a56b45d7a9b73a12 P 4d190acabedb48b3bbb8cd6b677622da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:49.228698 18169 raft_consensus.cc:388] T 2bfc0209bd434fa9a56b45d7a9b73a12 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:49.228823 18169 raft_consensus.cc:482] T 2bfc0209bd434fa9a56b45d7a9b73a12 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:49.228969 18169 raft_consensus.cc:3037] T 2bfc0209bd434fa9a56b45d7a9b73a12 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:49.232894 18169 raft_consensus.cc:504] T 2bfc0209bd434fa9a56b45d7a9b73a12 P 4d190acabedb48b3bbb8cd6b677622da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:49.233206 18169 leader_election.cc:302] T 2bfc0209bd434fa9a56b45d7a9b73a12 P 4d190acabedb48b3bbb8cd6b677622da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d190acabedb48b3bbb8cd6b677622da; no voters:
I20240502 01:52:49.233453 18169 leader_election.cc:288] T 2bfc0209bd434fa9a56b45d7a9b73a12 P 4d190acabedb48b3bbb8cd6b677622da [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:49.233673 18174 raft_consensus.cc:2781] T 2bfc0209bd434fa9a56b45d7a9b73a12 P 4d190acabedb48b3bbb8cd6b677622da [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:49.233913 18169 ts_tablet_manager.cc:1398] T 2bfc0209bd434fa9a56b45d7a9b73a12 P 4d190acabedb48b3bbb8cd6b677622da: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 01:52:49.234480 18155 raft_consensus.cc:686] T c130cb706b004c2088925fba192a8c90 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Becoming Leader. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Running, Role: LEADER
I20240502 01:52:49.234872 18174 raft_consensus.cc:686] T 2bfc0209bd434fa9a56b45d7a9b73a12 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Becoming Leader. State: Replica: 4d190acabedb48b3bbb8cd6b677622da, State: Running, Role: LEADER
I20240502 01:52:49.234903 18155 consensus_queue.cc:237] T c130cb706b004c2088925fba192a8c90 P b8ffc52e71f045e5ac0f8aa35f3eea38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:49.235306 18174 consensus_queue.cc:237] T 2bfc0209bd434fa9a56b45d7a9b73a12 P 4d190acabedb48b3bbb8cd6b677622da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:49.237593 17084 catalog_manager.cc:5484] T c130cb706b004c2088925fba192a8c90 P b8ffc52e71f045e5ac0f8aa35f3eea38 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } }
I20240502 01:52:49.240447 17085 catalog_manager.cc:5484] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194). New cstate: current_term: 1 leader_uuid: "4d190acabedb48b3bbb8cd6b677622da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } }
I20240502 01:52:49.245339 17085 catalog_manager.cc:5484] T 2bfc0209bd434fa9a56b45d7a9b73a12 P 4d190acabedb48b3bbb8cd6b677622da reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194). New cstate: current_term: 1 leader_uuid: "4d190acabedb48b3bbb8cd6b677622da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } }
W20240502 01:52:49.586778 18181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:49.587152 18181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:49.594926 18181 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 125.287 ms
time per row: 0.0626434 ms
I20240502 01:52:49.866107 17085 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53564:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:52:49.875386 17208 tablet_service.cc:1467] Processing CreateTablet for tablet e4a22d109d8044adaeb364037e9c137e (DEFAULT_TABLE table=rebalance_test_table_2 [id=c5068995c1864beca8f4cdbaff1fc3f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:49.875614 17209 tablet_service.cc:1467] Processing CreateTablet for tablet bf441bb1e6064b56b745ffe8d02a3983 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c5068995c1864beca8f4cdbaff1fc3f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:49.875911 17208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a22d109d8044adaeb364037e9c137e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:49.876108 17209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf441bb1e6064b56b745ffe8d02a3983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:49.877784 17342 tablet_service.cc:1467] Processing CreateTablet for tablet b1636e2cd56f40359df64d2ac4c9d806 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c5068995c1864beca8f4cdbaff1fc3f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:49.878248 17342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1636e2cd56f40359df64d2ac4c9d806. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:49.881992 18222 tablet_bootstrap.cc:492] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38: Bootstrap starting.
I20240502 01:52:49.883383 18222 tablet_bootstrap.cc:654] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:49.884267 18223 tablet_bootstrap.cc:492] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da: Bootstrap starting.
I20240502 01:52:49.885675 18222 tablet_bootstrap.cc:492] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38: No bootstrap required, opened a new log
I20240502 01:52:49.885824 18222 ts_tablet_manager.cc:1367] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:52:49.886159 18223 tablet_bootstrap.cc:654] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:49.886675 18222 raft_consensus.cc:348] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:49.886879 18222 raft_consensus.cc:374] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:49.886983 18222 raft_consensus.cc:729] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Initialized, Role: FOLLOWER
I20240502 01:52:49.887280 18222 consensus_queue.cc:260] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:49.887449 18222 raft_consensus.cc:388] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:49.887530 18222 raft_consensus.cc:482] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:49.887655 18222 raft_consensus.cc:3037] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:49.888249 18223 tablet_bootstrap.cc:492] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da: No bootstrap required, opened a new log
I20240502 01:52:49.888386 18223 ts_tablet_manager.cc:1367] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:52:49.889227 18223 raft_consensus.cc:348] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:49.889411 18223 raft_consensus.cc:374] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:49.889504 18223 raft_consensus.cc:729] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d190acabedb48b3bbb8cd6b677622da, State: Initialized, Role: FOLLOWER
I20240502 01:52:49.889776 18223 consensus_queue.cc:260] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:49.889974 18223 raft_consensus.cc:388] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:49.890056 18223 raft_consensus.cc:482] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:49.890168 18223 raft_consensus.cc:3037] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:49.890919 18222 raft_consensus.cc:504] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:49.891186 18222 leader_election.cc:302] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8ffc52e71f045e5ac0f8aa35f3eea38; no voters:
I20240502 01:52:49.891393 18222 leader_election.cc:288] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:49.891469 18155 raft_consensus.cc:2781] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:49.892024 18155 raft_consensus.cc:686] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Becoming Leader. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Running, Role: LEADER
I20240502 01:52:49.892293 18155 consensus_queue.cc:237] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:49.893330 18223 raft_consensus.cc:504] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:49.893730 18223 leader_election.cc:302] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d190acabedb48b3bbb8cd6b677622da; no voters:
I20240502 01:52:49.894145 18178 raft_consensus.cc:2781] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:49.894296 18178 raft_consensus.cc:686] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Becoming Leader. State: Replica: 4d190acabedb48b3bbb8cd6b677622da, State: Running, Role: LEADER
I20240502 01:52:49.894651 18178 consensus_queue.cc:237] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:49.895084 17085 catalog_manager.cc:5484] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } }
I20240502 01:52:49.897509 18223 leader_election.cc:288] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:49.897578 17082 catalog_manager.cc:5484] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194). New cstate: current_term: 1 leader_uuid: "4d190acabedb48b3bbb8cd6b677622da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } }
I20240502 01:52:49.898226 18222 ts_tablet_manager.cc:1398] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20240502 01:52:49.898411 18223 ts_tablet_manager.cc:1398] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240502 01:52:49.898569 18222 tablet_bootstrap.cc:492] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38: Bootstrap starting.
I20240502 01:52:49.900259 18222 tablet_bootstrap.cc:654] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:49.902923 18222 tablet_bootstrap.cc:492] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38: No bootstrap required, opened a new log
I20240502 01:52:49.903079 18222 ts_tablet_manager.cc:1367] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 01:52:49.903659 18222 raft_consensus.cc:348] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:49.903813 18222 raft_consensus.cc:374] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:49.903894 18222 raft_consensus.cc:729] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Initialized, Role: FOLLOWER
I20240502 01:52:49.904161 18222 consensus_queue.cc:260] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:49.904369 18222 raft_consensus.cc:388] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:49.904484 18222 raft_consensus.cc:482] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:49.904560 18222 raft_consensus.cc:3037] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:49.906500 18222 raft_consensus.cc:504] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:49.906673 18222 leader_election.cc:302] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8ffc52e71f045e5ac0f8aa35f3eea38; no voters:
I20240502 01:52:49.906873 18222 leader_election.cc:288] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:49.907013 18226 raft_consensus.cc:2781] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:49.907320 18226 raft_consensus.cc:686] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Becoming Leader. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Running, Role: LEADER
I20240502 01:52:49.907375 18222 ts_tablet_manager.cc:1398] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20240502 01:52:49.907734 18226 consensus_queue.cc:237] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:49.909909 17085 catalog_manager.cc:5484] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } }
W20240502 01:52:50.263913 18232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:50.264267 18232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:50.272408 18232 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 145.059 ms
time per row: 0.0725295 ms
I20240502 01:52:50.563148 17082 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53564:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:52:50.571590 17342 tablet_service.cc:1467] Processing CreateTablet for tablet 3fed2519c9b34871b27e5148e18b0673 (DEFAULT_TABLE table=rebalance_test_table_3 [id=08ac3856c4004b1ba9081159555cd945]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:50.571770 17341 tablet_service.cc:1467] Processing CreateTablet for tablet ba6e000d044241edbb347c80dca80b58 (DEFAULT_TABLE table=rebalance_test_table_3 [id=08ac3856c4004b1ba9081159555cd945]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:50.572134 17342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fed2519c9b34871b27e5148e18b0673. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:50.572026 17209 tablet_service.cc:1467] Processing CreateTablet for tablet 037045b61419472089771a8c6a057d21 (DEFAULT_TABLE table=rebalance_test_table_3 [id=08ac3856c4004b1ba9081159555cd945]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:50.572391 17341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6e000d044241edbb347c80dca80b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:50.572438 17209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 037045b61419472089771a8c6a057d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:50.578300 18280 tablet_bootstrap.cc:492] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da: Bootstrap starting.
I20240502 01:52:50.579468 18281 tablet_bootstrap.cc:492] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38: Bootstrap starting.
I20240502 01:52:50.580376 18280 tablet_bootstrap.cc:654] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:50.581509 18281 tablet_bootstrap.cc:654] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:50.582479 18280 tablet_bootstrap.cc:492] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da: No bootstrap required, opened a new log
I20240502 01:52:50.582615 18280 ts_tablet_manager.cc:1367] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:52:50.583189 18280 raft_consensus.cc:348] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:50.583340 18280 raft_consensus.cc:374] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:50.583379 18281 tablet_bootstrap.cc:492] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38: No bootstrap required, opened a new log
I20240502 01:52:50.583436 18280 raft_consensus.cc:729] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d190acabedb48b3bbb8cd6b677622da, State: Initialized, Role: FOLLOWER
I20240502 01:52:50.583519 18281 ts_tablet_manager.cc:1367] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:52:50.583683 18280 consensus_queue.cc:260] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:50.583843 18280 raft_consensus.cc:388] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:50.583925 18280 raft_consensus.cc:482] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:50.584039 18280 raft_consensus.cc:3037] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:50.584244 18281 raft_consensus.cc:348] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:50.584385 18281 raft_consensus.cc:374] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:50.584478 18281 raft_consensus.cc:729] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Initialized, Role: FOLLOWER
I20240502 01:52:50.584712 18281 consensus_queue.cc:260] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:50.584923 18281 raft_consensus.cc:388] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:50.585016 18281 raft_consensus.cc:482] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:50.585135 18281 raft_consensus.cc:3037] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:50.586220 18280 raft_consensus.cc:504] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:50.586449 18280 leader_election.cc:302] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d190acabedb48b3bbb8cd6b677622da; no voters:
I20240502 01:52:50.586628 18280 leader_election.cc:288] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:50.586714 18266 raft_consensus.cc:2781] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:50.588256 18281 raft_consensus.cc:504] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:50.588543 18281 leader_election.cc:302] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8ffc52e71f045e5ac0f8aa35f3eea38; no voters:
I20240502 01:52:50.588761 18281 leader_election.cc:288] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:50.588945 18270 raft_consensus.cc:2781] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:50.589316 18281 ts_tablet_manager.cc:1398] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20240502 01:52:50.590627 18280 ts_tablet_manager.cc:1398] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20240502 01:52:50.590946 18280 tablet_bootstrap.cc:492] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da: Bootstrap starting.
I20240502 01:52:50.591646 18266 raft_consensus.cc:686] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Becoming Leader. State: Replica: 4d190acabedb48b3bbb8cd6b677622da, State: Running, Role: LEADER
I20240502 01:52:50.592010 18266 consensus_queue.cc:237] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:50.592726 18280 tablet_bootstrap.cc:654] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:50.595999 18280 tablet_bootstrap.cc:492] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da: No bootstrap required, opened a new log
I20240502 01:52:50.596210 18280 ts_tablet_manager.cc:1367] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:52:50.596083 17082 catalog_manager.cc:5484] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194). New cstate: current_term: 1 leader_uuid: "4d190acabedb48b3bbb8cd6b677622da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } }
I20240502 01:52:50.597232 18280 raft_consensus.cc:348] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:50.597465 18280 raft_consensus.cc:374] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:50.597548 18280 raft_consensus.cc:729] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d190acabedb48b3bbb8cd6b677622da, State: Initialized, Role: FOLLOWER
I20240502 01:52:50.597831 18280 consensus_queue.cc:260] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:50.598860 18280 raft_consensus.cc:388] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:50.599028 18280 raft_consensus.cc:482] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:50.599172 18280 raft_consensus.cc:3037] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:50.601183 18270 raft_consensus.cc:686] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Becoming Leader. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Running, Role: LEADER
I20240502 01:52:50.601544 18270 consensus_queue.cc:237] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:50.602442 18280 raft_consensus.cc:504] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:50.602669 18280 leader_election.cc:302] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d190acabedb48b3bbb8cd6b677622da; no voters:
I20240502 01:52:50.602874 18280 leader_election.cc:288] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:50.602916 18266 raft_consensus.cc:2781] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:50.603359 18280 ts_tablet_manager.cc:1398] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 01:52:50.603601 18266 raft_consensus.cc:686] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Becoming Leader. State: Replica: 4d190acabedb48b3bbb8cd6b677622da, State: Running, Role: LEADER
I20240502 01:52:50.603818 18266 consensus_queue.cc:237] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:50.604087 17082 catalog_manager.cc:5484] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } }
I20240502 01:52:50.606745 17080 catalog_manager.cc:5484] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194). New cstate: current_term: 1 leader_uuid: "4d190acabedb48b3bbb8cd6b677622da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } }
W20240502 01:52:50.981356 18291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:50.981698 18291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:50.989395 18291 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 113.2 ms
time per row: 0.0565998 ms
I20240502 01:52:51.246332 17084 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53564:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:52:51.255354 17209 tablet_service.cc:1467] Processing CreateTablet for tablet 149a3a4584fe4c82a59ca4bc28ca80dd (DEFAULT_TABLE table=rebalance_test_table_4 [id=d01218100b4444e98fec7c8375220687]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.255492 17342 tablet_service.cc:1467] Processing CreateTablet for tablet 97ffbdb149fb42c892ed094ce60f7166 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d01218100b4444e98fec7c8375220687]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.255877 17342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ffbdb149fb42c892ed094ce60f7166. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.255887 17209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 149a3a4584fe4c82a59ca4bc28ca80dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.256070 17208 tablet_service.cc:1467] Processing CreateTablet for tablet bc1d0f4745df4203848a1f4820f9b499 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d01218100b4444e98fec7c8375220687]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.256421 17208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1d0f4745df4203848a1f4820f9b499. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.262625 18338 tablet_bootstrap.cc:492] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38: Bootstrap starting.
I20240502 01:52:51.263288 18339 tablet_bootstrap.cc:492] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da: Bootstrap starting.
I20240502 01:52:51.264308 18338 tablet_bootstrap.cc:654] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.264758 18339 tablet_bootstrap.cc:654] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.266434 18338 tablet_bootstrap.cc:492] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38: No bootstrap required, opened a new log
I20240502 01:52:51.266577 18338 ts_tablet_manager.cc:1367] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:52:51.266669 18339 tablet_bootstrap.cc:492] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da: No bootstrap required, opened a new log
I20240502 01:52:51.266819 18339 ts_tablet_manager.cc:1367] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:52:51.267266 18338 raft_consensus.cc:348] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:51.267422 18338 raft_consensus.cc:374] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.267406 18339 raft_consensus.cc:348] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:51.267520 18338 raft_consensus.cc:729] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.267593 18339 raft_consensus.cc:374] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.267688 18339 raft_consensus.cc:729] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d190acabedb48b3bbb8cd6b677622da, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.267768 18338 consensus_queue.cc:260] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:51.267892 18339 consensus_queue.cc:260] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:51.267968 18338 raft_consensus.cc:388] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:51.268045 18338 raft_consensus.cc:482] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:51.268066 18339 raft_consensus.cc:388] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:51.268138 18339 raft_consensus.cc:482] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:51.268157 18338 raft_consensus.cc:3037] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:51.268246 18339 raft_consensus.cc:3037] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:51.270385 18338 raft_consensus.cc:504] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:51.270427 18339 raft_consensus.cc:504] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:51.270653 18338 leader_election.cc:302] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8ffc52e71f045e5ac0f8aa35f3eea38; no voters:
I20240502 01:52:51.270664 18339 leader_election.cc:302] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d190acabedb48b3bbb8cd6b677622da; no voters:
I20240502 01:52:51.270850 18339 leader_election.cc:288] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:51.270933 18325 raft_consensus.cc:2781] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:51.270937 18328 raft_consensus.cc:2781] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:51.271415 18339 ts_tablet_manager.cc:1398] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:52:51.271409 18325 raft_consensus.cc:686] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Becoming Leader. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Running, Role: LEADER
I20240502 01:52:51.271764 18325 consensus_queue.cc:237] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:51.271874 18328 raft_consensus.cc:686] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Becoming Leader. State: Replica: 4d190acabedb48b3bbb8cd6b677622da, State: Running, Role: LEADER
I20240502 01:52:51.272174 18328 consensus_queue.cc:237] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:51.274039 18338 leader_election.cc:288] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:51.274497 18338 ts_tablet_manager.cc:1398] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20240502 01:52:51.274785 18338 tablet_bootstrap.cc:492] T 149a3a4584fe4c82a59ca4bc28ca80dd P b8ffc52e71f045e5ac0f8aa35f3eea38: Bootstrap starting.
I20240502 01:52:51.274772 17080 catalog_manager.cc:5484] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194). New cstate: current_term: 1 leader_uuid: "4d190acabedb48b3bbb8cd6b677622da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } }
I20240502 01:52:51.275832 17084 catalog_manager.cc:5484] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } }
I20240502 01:52:51.276652 18338 tablet_bootstrap.cc:654] T 149a3a4584fe4c82a59ca4bc28ca80dd P b8ffc52e71f045e5ac0f8aa35f3eea38: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.279836 18338 tablet_bootstrap.cc:492] T 149a3a4584fe4c82a59ca4bc28ca80dd P b8ffc52e71f045e5ac0f8aa35f3eea38: No bootstrap required, opened a new log
I20240502 01:52:51.280030 18338 ts_tablet_manager.cc:1367] T 149a3a4584fe4c82a59ca4bc28ca80dd P b8ffc52e71f045e5ac0f8aa35f3eea38: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 01:52:51.280896 18338 raft_consensus.cc:348] T 149a3a4584fe4c82a59ca4bc28ca80dd P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:51.281100 18338 raft_consensus.cc:374] T 149a3a4584fe4c82a59ca4bc28ca80dd P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.281189 18338 raft_consensus.cc:729] T 149a3a4584fe4c82a59ca4bc28ca80dd P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.281440 18338 consensus_queue.cc:260] T 149a3a4584fe4c82a59ca4bc28ca80dd P b8ffc52e71f045e5ac0f8aa35f3eea38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:51.281623 18338 raft_consensus.cc:388] T 149a3a4584fe4c82a59ca4bc28ca80dd P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:51.281735 18338 raft_consensus.cc:482] T 149a3a4584fe4c82a59ca4bc28ca80dd P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:51.281852 18338 raft_consensus.cc:3037] T 149a3a4584fe4c82a59ca4bc28ca80dd P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:51.284606 18338 raft_consensus.cc:504] T 149a3a4584fe4c82a59ca4bc28ca80dd P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:51.284911 18338 leader_election.cc:302] T 149a3a4584fe4c82a59ca4bc28ca80dd P b8ffc52e71f045e5ac0f8aa35f3eea38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8ffc52e71f045e5ac0f8aa35f3eea38; no voters:
I20240502 01:52:51.285130 18338 leader_election.cc:288] T 149a3a4584fe4c82a59ca4bc28ca80dd P b8ffc52e71f045e5ac0f8aa35f3eea38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:51.285213 18344 raft_consensus.cc:2781] T 149a3a4584fe4c82a59ca4bc28ca80dd P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:51.285604 18338 ts_tablet_manager.cc:1398] T 149a3a4584fe4c82a59ca4bc28ca80dd P b8ffc52e71f045e5ac0f8aa35f3eea38: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20240502 01:52:51.285810 18344 raft_consensus.cc:686] T 149a3a4584fe4c82a59ca4bc28ca80dd P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Becoming Leader. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Running, Role: LEADER
I20240502 01:52:51.286177 18344 consensus_queue.cc:237] T 149a3a4584fe4c82a59ca4bc28ca80dd P b8ffc52e71f045e5ac0f8aa35f3eea38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:51.288484 17078 catalog_manager.cc:5484] T 149a3a4584fe4c82a59ca4bc28ca80dd P b8ffc52e71f045e5ac0f8aa35f3eea38 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } }
W20240502 01:52:51.636504 18350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:51.636857 18350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:51.647483 18350 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 120.771 ms
time per row: 0.0603853 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:52:52.838703 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.195:42281
--local_ip_for_outbound_sockets=127.14.212.195
--tserver_master_addrs=127.14.212.254:43511
--webserver_port=39119
--webserver_interface=127.14.212.195
--builtin_ntp_servers=127.14.212.212:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:52:53.177686 18391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:53.177997 18391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:53.178125 18391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:53.186172 18391 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:52:53.186313 18391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:53.186486 18391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.195
I20240502 01:52:53.198655 18391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.195:42281
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.212.195
--webserver_port=39119
--tserver_master_addrs=127.14.212.254:43511
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.195
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:53.200028 18391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:53.201828 18391 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:53.215039 18397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:53.215644 18396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:53.216305 18399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:53.217234 18391 server_base.cc:1034] running on GCE node
I20240502 01:52:53.217965 18391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:53.219010 18391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:53.220250 18391 hybrid_clock.cc:648] HybridClock initialized: now 1714614773220206 us; error 51 us; skew 500 ppm
I20240502 01:52:53.220609 18391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:53.223237 18391 webserver.cc:460] Webserver started at http://127.14.212.195:39119/ using document root <none> and password file <none>
I20240502 01:52:53.224040 18391 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:53.224179 18391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:53.230055 18391 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 01:52:53.233269 18405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:53.234737 18391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:52:53.235010 18391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75"
format_stamp: "Formatted at 2024-05-02 01:52:42 on dist-test-slave-mk1r"
I20240502 01:52:53.235931 18391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:53.255730 18391 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:53.256670 18391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:53.257009 18391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:53.258149 18391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:53.260177 18412 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 01:52:53.266115 18391 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 01:52:53.266232 18391 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20240502 01:52:53.266368 18391 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 01:52:53.269500 18391 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 01:52:53.269603 18391 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:53.270010 18412 tablet_bootstrap.cc:492] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75: Bootstrap starting.
I20240502 01:52:53.291245 18412 log.cc:824] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:53.296450 18412 tablet_bootstrap.cc:492] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:52:53.297219 18412 tablet_bootstrap.cc:492] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75: Bootstrap complete.
I20240502 01:52:53.298910 18412 ts_tablet_manager.cc:1367] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.013s
I20240502 01:52:53.309733 18412 raft_consensus.cc:348] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } }
I20240502 01:52:53.310911 18412 raft_consensus.cc:729] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e4a5c2ba664cd08c3aba1bb348ed75, State: Initialized, Role: FOLLOWER
I20240502 01:52:53.311774 18412 consensus_queue.cc:260] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } }
I20240502 01:52:53.311983 18412 raft_consensus.cc:388] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:53.312116 18412 raft_consensus.cc:482] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:53.312287 18412 raft_consensus.cc:3037] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:53.314287 18391 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.195:42281
I20240502 01:52:53.314359 18521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.195:42281 every 8 connection(s)
I20240502 01:52:53.315742 18391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:52:53.316980 18412 raft_consensus.cc:504] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } }
I20240502 01:52:53.317592 18412 leader_election.cc:302] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4e4a5c2ba664cd08c3aba1bb348ed75; no voters:
I20240502 01:52:53.318295 18412 leader_election.cc:288] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240502 01:52:53.319183 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 18391
I20240502 01:52:53.319777 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.196:43697
--local_ip_for_outbound_sockets=127.14.212.196
--tserver_master_addrs=127.14.212.254:43511
--webserver_port=40389
--webserver_interface=127.14.212.196
--builtin_ntp_servers=127.14.212.212:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:52:53.325158 18524 raft_consensus.cc:2781] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:53.331665 18524 raft_consensus.cc:686] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 2 LEADER]: Becoming Leader. State: Replica: d4e4a5c2ba664cd08c3aba1bb348ed75, State: Running, Role: LEADER
I20240502 01:52:53.332466 18412 ts_tablet_manager.cc:1398] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75: Time spent starting tablet: real 0.033s user 0.018s sys 0.014s
I20240502 01:52:53.332415 18524 consensus_queue.cc:237] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } }
I20240502 01:52:53.334646 18522 heartbeater.cc:347] Connected to a master server at 127.14.212.254:43511
I20240502 01:52:53.334967 18522 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:53.335947 18522 heartbeater.cc:511] Master 127.14.212.254:43511 requested a full tablet report, sending...
I20240502 01:52:53.343971 17084 ts_manager.cc:194] Re-registered known tserver with Master: d4e4a5c2ba664cd08c3aba1bb348ed75 (127.14.212.195:42281)
I20240502 01:52:53.344404 17084 catalog_manager.cc:5484] T a487f57191fe4eb998ad8f5f86f4050b P d4e4a5c2ba664cd08c3aba1bb348ed75 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } } }
I20240502 01:52:53.349416 17084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.195:40483
I20240502 01:52:53.352559 18522 heartbeater.cc:503] Master 127.14.212.254:43511 was elected leader, sending a full tablet report...
W20240502 01:52:53.671623 18528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:53.671943 18528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:53.672046 18528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:53.679709 18528 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:52:53.679849 18528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:53.679968 18528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.196
I20240502 01:52:53.689226 18528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.196:43697
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.212.196
--webserver_port=40389
--tserver_master_addrs=127.14.212.254:43511
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.196
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:53.690531 18528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:53.692375 18528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:53.705073 18540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:53.705678 18541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:53.706198 18543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:53.706950 18528 server_base.cc:1034] running on GCE node
I20240502 01:52:53.707573 18528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:53.708599 18528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:53.709847 18528 hybrid_clock.cc:648] HybridClock initialized: now 1714614773709767 us; error 68 us; skew 500 ppm
I20240502 01:52:53.710211 18528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:53.712599 18528 webserver.cc:460] Webserver started at http://127.14.212.196:40389/ using document root <none> and password file <none>
I20240502 01:52:53.713403 18528 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:53.713538 18528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:53.718889 18528 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:52:53.721825 18549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:53.723277 18528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:53.723448 18528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "d21ca31c11b746d99a102d116cfad615"
format_stamp: "Formatted at 2024-05-02 01:52:42 on dist-test-slave-mk1r"
I20240502 01:52:53.724323 18528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:53.755961 18528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:53.756832 18528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:53.757155 18528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:53.758157 18528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:53.759784 18528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:53.759867 18528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:53.760010 18528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:53.760066 18528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:53.793210 18528 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.196:43697
I20240502 01:52:53.793267 18664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.196:43697 every 8 connection(s)
I20240502 01:52:53.794693 18528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:52:53.803400 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 18528
I20240502 01:52:53.804212 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.197:41667
--local_ip_for_outbound_sockets=127.14.212.197
--tserver_master_addrs=127.14.212.254:43511
--webserver_port=36535
--webserver_interface=127.14.212.197
--builtin_ntp_servers=127.14.212.212:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:52:53.807812 18665 heartbeater.cc:347] Connected to a master server at 127.14.212.254:43511
I20240502 01:52:53.808161 18665 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:53.809128 18665 heartbeater.cc:511] Master 127.14.212.254:43511 requested a full tablet report, sending...
I20240502 01:52:53.810781 17084 ts_manager.cc:194] Re-registered known tserver with Master: d21ca31c11b746d99a102d116cfad615 (127.14.212.196:43697)
I20240502 01:52:53.812036 17084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.196:52109
W20240502 01:52:54.146836 18668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:54.147166 18668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:54.147264 18668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:54.155262 18668 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:52:54.155417 18668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:54.155541 18668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.197
I20240502 01:52:54.165479 18668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.197:41667
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.212.197
--webserver_port=36535
--tserver_master_addrs=127.14.212.254:43511
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.197
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:54.166851 18668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:54.168697 18668 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:54.181515 18676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:54.182129 18673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:54.182821 18674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:54.183686 18668 server_base.cc:1034] running on GCE node
I20240502 01:52:54.184341 18668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:54.185421 18668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:54.186659 18668 hybrid_clock.cc:648] HybridClock initialized: now 1714614774186610 us; error 47 us; skew 500 ppm
I20240502 01:52:54.186993 18668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:54.194273 18668 webserver.cc:460] Webserver started at http://127.14.212.197:36535/ using document root <none> and password file <none>
I20240502 01:52:54.195075 18668 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:54.195212 18668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:54.200664 18668 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:52:54.203538 18682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:54.205004 18668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:54.205169 18668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "adf51a82733d4010b3028cb54cc307e2"
format_stamp: "Formatted at 2024-05-02 01:52:42 on dist-test-slave-mk1r"
I20240502 01:52:54.206007 18668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:54.221076 18668 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:54.221961 18668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:54.222277 18668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:54.223302 18668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:54.224934 18668 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:54.225037 18668 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:54.225174 18668 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:54.225242 18668 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:54.259023 18668 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.197:41667
I20240502 01:52:54.259101 18796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.197:41667 every 8 connection(s)
I20240502 01:52:54.260594 18668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:52:54.265667 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 18668
I20240502 01:52:54.266482 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.198:34487
--local_ip_for_outbound_sockets=127.14.212.198
--tserver_master_addrs=127.14.212.254:43511
--webserver_port=42501
--webserver_interface=127.14.212.198
--builtin_ntp_servers=127.14.212.212:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:52:54.275802 18797 heartbeater.cc:347] Connected to a master server at 127.14.212.254:43511
I20240502 01:52:54.276180 18797 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:54.277094 18797 heartbeater.cc:511] Master 127.14.212.254:43511 requested a full tablet report, sending...
I20240502 01:52:54.278770 17084 ts_manager.cc:194] Re-registered known tserver with Master: adf51a82733d4010b3028cb54cc307e2 (127.14.212.197:41667)
I20240502 01:52:54.280045 17084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.197:32861
W20240502 01:52:54.612983 18800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:54.613305 18800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:54.613410 18800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:54.621240 18800 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:52:54.621404 18800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:54.621618 18800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.198
I20240502 01:52:54.631536 18800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.198:34487
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.14.212.198
--webserver_port=42501
--tserver_master_addrs=127.14.212.254:43511
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.198
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:54.632995 18800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:54.634974 18800 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:54.648569 18805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:54.649407 18808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:54.650027 18806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:54.650825 18800 server_base.cc:1034] running on GCE node
I20240502 01:52:54.651460 18800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:54.652472 18800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:54.653702 18800 hybrid_clock.cc:648] HybridClock initialized: now 1714614774653649 us; error 57 us; skew 500 ppm
I20240502 01:52:54.654083 18800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:54.656630 18800 webserver.cc:460] Webserver started at http://127.14.212.198:42501/ using document root <none> and password file <none>
I20240502 01:52:54.657471 18800 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:54.657604 18800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:54.663415 18800 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:52:54.666445 18814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:54.667954 18800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 01:52:54.668116 18800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a"
format_stamp: "Formatted at 2024-05-02 01:52:43 on dist-test-slave-mk1r"
I20240502 01:52:54.669063 18800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:54.698179 18800 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:54.699098 18800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:54.699407 18800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:54.700443 18800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:54.702128 18800 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:54.702209 18800 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:54.702333 18800 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:54.702384 18800 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:54.738277 18800 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.198:34487
I20240502 01:52:54.738340 18928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.198:34487 every 8 connection(s)
I20240502 01:52:54.739836 18800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:52:54.748980 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 18800
I20240502 01:52:54.749780 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.199:45747
--local_ip_for_outbound_sockets=127.14.212.199
--tserver_master_addrs=127.14.212.254:43511
--webserver_port=41373
--webserver_interface=127.14.212.199
--builtin_ntp_servers=127.14.212.212:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:52:54.754047 18929 heartbeater.cc:347] Connected to a master server at 127.14.212.254:43511
I20240502 01:52:54.754415 18929 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:54.755288 18929 heartbeater.cc:511] Master 127.14.212.254:43511 requested a full tablet report, sending...
I20240502 01:52:54.756985 17084 ts_manager.cc:194] Re-registered known tserver with Master: aaf6d9dc6f5747bd83d2b9e24a7d679a (127.14.212.198:34487)
I20240502 01:52:54.758276 17084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.198:40293
I20240502 01:52:54.814708 18665 heartbeater.cc:503] Master 127.14.212.254:43511 was elected leader, sending a full tablet report...
W20240502 01:52:55.151142 18932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:55.151556 18932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:55.151762 18932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:55.162797 18932 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:52:55.162997 18932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:55.163192 18932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.199
I20240502 01:52:55.175920 18932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.199:45747
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.14.212.199
--webserver_port=41373
--tserver_master_addrs=127.14.212.254:43511
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.199
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:55.177877 18932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:55.180509 18932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:55.199143 18938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:55.199785 18941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:55.200400 18939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:55.201061 18932 server_base.cc:1034] running on GCE node
I20240502 01:52:55.201676 18932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:55.202692 18932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:55.203891 18932 hybrid_clock.cc:648] HybridClock initialized: now 1714614775203850 us; error 38 us; skew 500 ppm
I20240502 01:52:55.204209 18932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:55.206723 18932 webserver.cc:460] Webserver started at http://127.14.212.199:41373/ using document root <none> and password file <none>
I20240502 01:52:55.207584 18932 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:55.207772 18932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:55.214409 18932 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 01:52:55.217887 18947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:55.219552 18932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:52:55.219782 18932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f7baee4920524734aa33ce500a559f3b"
format_stamp: "Formatted at 2024-05-02 01:52:43 on dist-test-slave-mk1r"
I20240502 01:52:55.220772 18932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:55.239493 18932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:55.240523 18932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:55.240960 18932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:55.242416 18932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:55.244541 18932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:55.244642 18932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:55.244812 18932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:55.244907 18932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:55.282941 18797 heartbeater.cc:503] Master 127.14.212.254:43511 was elected leader, sending a full tablet report...
I20240502 01:52:55.296563 18932 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.199:45747
I20240502 01:52:55.296679 19061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.199:45747 every 8 connection(s)
I20240502 01:52:55.298288 18932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:52:55.303041 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 18932
I20240502 01:52:55.313797 19062 heartbeater.cc:347] Connected to a master server at 127.14.212.254:43511
I20240502 01:52:55.314230 19062 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:55.315737 19062 heartbeater.cc:511] Master 127.14.212.254:43511 requested a full tablet report, sending...
I20240502 01:52:55.317483 17084 ts_manager.cc:194] Re-registered known tserver with Master: f7baee4920524734aa33ce500a559f3b (127.14.212.199:45747)
I20240502 01:52:55.318745 17084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.199:42439
I20240502 01:52:55.761135 18929 heartbeater.cc:503] Master 127.14.212.254:43511 was elected leader, sending a full tablet report...
I20240502 01:52:55.907804 17342 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:52:55.916043 18730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:55.926537 18864 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:55.962052 18598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:55.964721 17209 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:52:55.985703 18455 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:55.996138 18996 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:56.174589 18864 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:56.177294 17342 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:52:56.183796 18730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:56.187687 17209 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:52:56.213907 18455 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:56.218890 18598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:56.268266 18996 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:56.321683 19062 heartbeater.cc:503] Master 127.14.212.254:43511 was elected leader, sending a full tablet report...
I20240502 01:52:56.359489 17230 consensus_queue.cc:237] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } }
W20240502 01:52:56.369433 17163 consensus_peers.cc:485] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 -> Peer d21ca31c11b746d99a102d116cfad615 (127.14.212.196:43697): Couldn't send request to peer d21ca31c11b746d99a102d116cfad615. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4a22d109d8044adaeb364037e9c137e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:56.941409 19166 ts_tablet_manager.cc:902] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615: Initiating tablet copy from peer b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193:44487)
I20240502 01:52:56.943414 19166 tablet_copy_client.cc:308] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.193:44487
I20240502 01:52:56.950423 17250 tablet_copy_service.cc:140] P b8ffc52e71f045e5ac0f8aa35f3eea38: Received BeginTabletCopySession request for tablet e4a22d109d8044adaeb364037e9c137e from peer d21ca31c11b746d99a102d116cfad615 ({username='slave'} at 127.14.212.196:53179)
I20240502 01:52:56.950733 17250 tablet_copy_service.cc:161] P b8ffc52e71f045e5ac0f8aa35f3eea38: Beginning new tablet copy session on tablet e4a22d109d8044adaeb364037e9c137e from peer d21ca31c11b746d99a102d116cfad615 at {username='slave'} at 127.14.212.196:53179: session id = d21ca31c11b746d99a102d116cfad615-e4a22d109d8044adaeb364037e9c137e
I20240502 01:52:56.953651 17250 tablet_copy_source_session.cc:205] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:56.958570 19166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a22d109d8044adaeb364037e9c137e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:56.970043 19166 tablet_copy_client.cc:783] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:56.970419 19166 tablet_copy_client.cc:647] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:56.973196 19166 tablet_copy_client.cc:523] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:52:56.976694 19166 tablet_bootstrap.cc:492] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615: Bootstrap starting.
I20240502 01:52:56.986789 19166 log.cc:824] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:57.019834 19166 tablet_bootstrap.cc:492] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:52:57.020480 19166 tablet_bootstrap.cc:492] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615: Bootstrap complete.
I20240502 01:52:57.021036 19166 ts_tablet_manager.cc:1367] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615: Time spent bootstrapping tablet: real 0.045s user 0.043s sys 0.005s
I20240502 01:52:57.026324 19166 raft_consensus.cc:348] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:57.027261 19166 raft_consensus.cc:729] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d21ca31c11b746d99a102d116cfad615, State: Initialized, Role: FOLLOWER
I20240502 01:52:57.027736 19166 consensus_queue.cc:260] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } }
I20240502 01:52:57.029560 19166 ts_tablet_manager.cc:1398] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 01:52:57.030974 17250 tablet_copy_service.cc:342] P b8ffc52e71f045e5ac0f8aa35f3eea38: Request end of tablet copy session d21ca31c11b746d99a102d116cfad615-e4a22d109d8044adaeb364037e9c137e received from {username='slave'} at 127.14.212.196:53179
I20240502 01:52:57.031195 17250 tablet_copy_service.cc:433] P b8ffc52e71f045e5ac0f8aa35f3eea38: ending tablet copy session d21ca31c11b746d99a102d116cfad615-e4a22d109d8044adaeb364037e9c137e on tablet e4a22d109d8044adaeb364037e9c137e with peer d21ca31c11b746d99a102d116cfad615
I20240502 01:52:57.396457 18619 raft_consensus.cc:1203] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:52:57.398610 19175 raft_consensus.cc:2932] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d21ca31c11b746d99a102d116cfad615 (127.14.212.196) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } } }
I20240502 01:52:57.399644 18619 raft_consensus.cc:2932] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d21ca31c11b746d99a102d116cfad615 (127.14.212.196) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } } }
I20240502 01:52:57.403617 19175 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:60962 (request call id 0) took 1045 ms. Trace:
I20240502 01:52:57.403913 19175 rpcz_store.cc:276] 0502 01:52:56.358366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 01:52:56.358430 (+ 64us) service_pool.cc:225] Handling call
0502 01:52:56.360789 (+ 2359us) log.cc:842] Serialized 263 byte log entry
0502 01:52:57.403515 (+1042726us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":84,"raft.queue_time_us":396,"raft.run_cpu_time_us":630,"raft.run_wall_time_us":630,"spinlock_wait_cycles":90240,"thread_start_us":669,"threads_started":2,"wal-append.queue_time_us":491,"wal-append.run_cpu_time_us":758,"wal-append.run_wall_time_us":1000810}
I20240502 01:52:57.405786 17078 catalog_manager.cc:5484] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615 reported cstate change: config changed from index -1 to 4, VOTER d21ca31c11b746d99a102d116cfad615 (127.14.212.196) added. New cstate: current_term: 1 leader_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } } }
I20240502 01:52:57.444737 17362 consensus_queue.cc:237] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } }
W20240502 01:52:57.455371 17298 consensus_peers.cc:485] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da -> Peer adf51a82733d4010b3028cb54cc307e2 (127.14.212.197:41667): Couldn't send request to peer adf51a82733d4010b3028cb54cc307e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fed2519c9b34871b27e5148e18b0673. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:57.927776 19203 ts_tablet_manager.cc:902] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2: Initiating tablet copy from peer 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194:34515)
I20240502 01:52:57.929809 19203 tablet_copy_client.cc:308] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:34515
I20240502 01:52:57.937095 17383 tablet_copy_service.cc:140] P 4d190acabedb48b3bbb8cd6b677622da: Received BeginTabletCopySession request for tablet 3fed2519c9b34871b27e5148e18b0673 from peer adf51a82733d4010b3028cb54cc307e2 ({username='slave'} at 127.14.212.197:41789)
I20240502 01:52:57.937346 17383 tablet_copy_service.cc:161] P 4d190acabedb48b3bbb8cd6b677622da: Beginning new tablet copy session on tablet 3fed2519c9b34871b27e5148e18b0673 from peer adf51a82733d4010b3028cb54cc307e2 at {username='slave'} at 127.14.212.197:41789: session id = adf51a82733d4010b3028cb54cc307e2-3fed2519c9b34871b27e5148e18b0673
I20240502 01:52:57.940065 17383 tablet_copy_source_session.cc:205] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:57.945466 19203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fed2519c9b34871b27e5148e18b0673. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.956830 19203 tablet_copy_client.cc:783] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:57.957288 19203 tablet_copy_client.cc:647] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:57.959901 19203 tablet_copy_client.cc:523] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:52:57.963420 19203 tablet_bootstrap.cc:492] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2: Bootstrap starting.
I20240502 01:52:57.973657 19203 log.cc:824] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:58.008255 19203 tablet_bootstrap.cc:492] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:52:58.008976 19203 tablet_bootstrap.cc:492] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2: Bootstrap complete.
I20240502 01:52:58.009421 19203 ts_tablet_manager.cc:1367] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2: Time spent bootstrapping tablet: real 0.046s user 0.040s sys 0.007s
I20240502 01:52:58.014379 19203 raft_consensus.cc:348] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:52:58.015408 19203 raft_consensus.cc:729] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf51a82733d4010b3028cb54cc307e2, State: Initialized, Role: FOLLOWER
I20240502 01:52:58.015862 19203 consensus_queue.cc:260] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } }
I20240502 01:52:58.017625 19203 ts_tablet_manager.cc:1398] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 01:52:58.018926 17383 tablet_copy_service.cc:342] P 4d190acabedb48b3bbb8cd6b677622da: Request end of tablet copy session adf51a82733d4010b3028cb54cc307e2-3fed2519c9b34871b27e5148e18b0673 received from {username='slave'} at 127.14.212.197:41789
I20240502 01:52:58.019145 17383 tablet_copy_service.cc:433] P 4d190acabedb48b3bbb8cd6b677622da: ending tablet copy session adf51a82733d4010b3028cb54cc307e2-3fed2519c9b34871b27e5148e18b0673 on tablet 3fed2519c9b34871b27e5148e18b0673 with peer adf51a82733d4010b3028cb54cc307e2
I20240502 01:52:58.366499 18751 raft_consensus.cc:1203] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:52:58.368659 19213 raft_consensus.cc:2932] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER adf51a82733d4010b3028cb54cc307e2 (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } } }
I20240502 01:52:58.369565 18751 raft_consensus.cc:2932] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER adf51a82733d4010b3028cb54cc307e2 (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } } }
I20240502 01:52:58.375885 17078 catalog_manager.cc:5484] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da reported cstate change: config changed from index -1 to 4, VOTER adf51a82733d4010b3028cb54cc307e2 (127.14.212.197) added. New cstate: current_term: 1 leader_uuid: "4d190acabedb48b3bbb8cd6b677622da" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } } }
I20240502 01:52:58.407812 17230 consensus_queue.cc:237] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } }
W20240502 01:52:58.416931 17162 consensus_peers.cc:485] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 -> Peer aaf6d9dc6f5747bd83d2b9e24a7d679a (127.14.212.198:34487): Couldn't send request to peer aaf6d9dc6f5747bd83d2b9e24a7d679a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e8092e428084a56bbfaaca963877631. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:58.953923 19241 ts_tablet_manager.cc:902] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a: Initiating tablet copy from peer b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193:44487)
I20240502 01:52:58.955917 19241 tablet_copy_client.cc:308] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.193:44487
I20240502 01:52:58.963188 17250 tablet_copy_service.cc:140] P b8ffc52e71f045e5ac0f8aa35f3eea38: Received BeginTabletCopySession request for tablet 1e8092e428084a56bbfaaca963877631 from peer aaf6d9dc6f5747bd83d2b9e24a7d679a ({username='slave'} at 127.14.212.198:42963)
I20240502 01:52:58.963394 17250 tablet_copy_service.cc:161] P b8ffc52e71f045e5ac0f8aa35f3eea38: Beginning new tablet copy session on tablet 1e8092e428084a56bbfaaca963877631 from peer aaf6d9dc6f5747bd83d2b9e24a7d679a at {username='slave'} at 127.14.212.198:42963: session id = aaf6d9dc6f5747bd83d2b9e24a7d679a-1e8092e428084a56bbfaaca963877631
I20240502 01:52:58.965457 17250 tablet_copy_source_session.cc:205] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:58.969931 19241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e8092e428084a56bbfaaca963877631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:58.981506 19241 tablet_copy_client.cc:783] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:58.981819 19241 tablet_copy_client.cc:647] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:58.984272 19241 tablet_copy_client.cc:523] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:52:58.987686 19241 tablet_bootstrap.cc:492] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a: Bootstrap starting.
I20240502 01:52:58.997826 19241 log.cc:824] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:59.031553 19241 tablet_bootstrap.cc:492] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:52:59.032239 19241 tablet_bootstrap.cc:492] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a: Bootstrap complete.
I20240502 01:52:59.032703 19241 ts_tablet_manager.cc:1367] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a: Time spent bootstrapping tablet: real 0.045s user 0.035s sys 0.008s
I20240502 01:52:59.037838 19241 raft_consensus.cc:348] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:52:59.038805 19241 raft_consensus.cc:729] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf6d9dc6f5747bd83d2b9e24a7d679a, State: Initialized, Role: FOLLOWER
I20240502 01:52:59.039315 19241 consensus_queue.cc:260] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } }
I20240502 01:52:59.041019 19241 ts_tablet_manager.cc:1398] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a: Time spent starting tablet: real 0.008s user 0.012s sys 0.000s
I20240502 01:52:59.042176 17250 tablet_copy_service.cc:342] P b8ffc52e71f045e5ac0f8aa35f3eea38: Request end of tablet copy session aaf6d9dc6f5747bd83d2b9e24a7d679a-1e8092e428084a56bbfaaca963877631 received from {username='slave'} at 127.14.212.198:42963
I20240502 01:52:59.042335 17250 tablet_copy_service.cc:433] P b8ffc52e71f045e5ac0f8aa35f3eea38: ending tablet copy session aaf6d9dc6f5747bd83d2b9e24a7d679a-1e8092e428084a56bbfaaca963877631 on tablet 1e8092e428084a56bbfaaca963877631 with peer aaf6d9dc6f5747bd83d2b9e24a7d679a
I20240502 01:52:59.392318 18884 raft_consensus.cc:1203] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:52:59.394282 19250 raft_consensus.cc:2932] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aaf6d9dc6f5747bd83d2b9e24a7d679a (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } } }
I20240502 01:52:59.395202 18884 raft_consensus.cc:2932] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aaf6d9dc6f5747bd83d2b9e24a7d679a (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } } }
I20240502 01:52:59.399960 17084 catalog_manager.cc:5484] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 reported cstate change: config changed from index -1 to 4, VOTER aaf6d9dc6f5747bd83d2b9e24a7d679a (127.14.212.198) added. New cstate: current_term: 1 leader_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } } }
I20240502 01:52:59.438905 17230 consensus_queue.cc:237] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } }
W20240502 01:52:59.448138 17163 consensus_peers.cc:485] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 -> Peer f7baee4920524734aa33ce500a559f3b (127.14.212.199:45747): Couldn't send request to peer f7baee4920524734aa33ce500a559f3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc1d0f4745df4203848a1f4820f9b499. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:59.970008 19276 ts_tablet_manager.cc:902] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b: Initiating tablet copy from peer b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193:44487)
I20240502 01:52:59.971956 19276 tablet_copy_client.cc:308] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.193:44487
I20240502 01:52:59.978912 17250 tablet_copy_service.cc:140] P b8ffc52e71f045e5ac0f8aa35f3eea38: Received BeginTabletCopySession request for tablet bc1d0f4745df4203848a1f4820f9b499 from peer f7baee4920524734aa33ce500a559f3b ({username='slave'} at 127.14.212.199:38983)
I20240502 01:52:59.979121 17250 tablet_copy_service.cc:161] P b8ffc52e71f045e5ac0f8aa35f3eea38: Beginning new tablet copy session on tablet bc1d0f4745df4203848a1f4820f9b499 from peer f7baee4920524734aa33ce500a559f3b at {username='slave'} at 127.14.212.199:38983: session id = f7baee4920524734aa33ce500a559f3b-bc1d0f4745df4203848a1f4820f9b499
I20240502 01:52:59.981292 17250 tablet_copy_source_session.cc:205] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:59.985909 19276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1d0f4745df4203848a1f4820f9b499. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:59.996753 19276 tablet_copy_client.cc:783] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:59.997145 19276 tablet_copy_client.cc:647] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:59.999575 19276 tablet_copy_client.cc:523] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:00.003047 19276 tablet_bootstrap.cc:492] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b: Bootstrap starting.
I20240502 01:53:00.013345 19276 log.cc:824] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:00.047039 19276 tablet_bootstrap.cc:492] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:00.047683 19276 tablet_bootstrap.cc:492] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b: Bootstrap complete.
I20240502 01:53:00.048095 19276 ts_tablet_manager.cc:1367] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b: Time spent bootstrapping tablet: real 0.045s user 0.039s sys 0.004s
I20240502 01:53:00.052841 19276 raft_consensus.cc:348] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:53:00.053925 19276 raft_consensus.cc:729] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7baee4920524734aa33ce500a559f3b, State: Initialized, Role: FOLLOWER
I20240502 01:53:00.054404 19276 consensus_queue.cc:260] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } }
I20240502 01:53:00.056139 19276 ts_tablet_manager.cc:1398] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b: Time spent starting tablet: real 0.008s user 0.006s sys 0.004s
I20240502 01:53:00.057271 17250 tablet_copy_service.cc:342] P b8ffc52e71f045e5ac0f8aa35f3eea38: Request end of tablet copy session f7baee4920524734aa33ce500a559f3b-bc1d0f4745df4203848a1f4820f9b499 received from {username='slave'} at 127.14.212.199:38983
I20240502 01:53:00.057426 17250 tablet_copy_service.cc:433] P b8ffc52e71f045e5ac0f8aa35f3eea38: ending tablet copy session f7baee4920524734aa33ce500a559f3b-bc1d0f4745df4203848a1f4820f9b499 on tablet bc1d0f4745df4203848a1f4820f9b499 with peer f7baee4920524734aa33ce500a559f3b
I20240502 01:53:00.399677 19016 raft_consensus.cc:1203] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:00.401659 19285 raft_consensus.cc:2932] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f7baee4920524734aa33ce500a559f3b (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } } }
I20240502 01:53:00.402556 19016 raft_consensus.cc:2932] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f7baee4920524734aa33ce500a559f3b (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } } }
I20240502 01:53:00.407891 17078 catalog_manager.cc:5484] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 reported cstate change: config changed from index -1 to 4, VOTER f7baee4920524734aa33ce500a559f3b (127.14.212.199) added. New cstate: current_term: 1 leader_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } } }
I20240502 01:53:00.440156 17362 consensus_queue.cc:237] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } }
W20240502 01:53:00.449198 17294 consensus_peers.cc:485] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da -> Peer d4e4a5c2ba664cd08c3aba1bb348ed75 (127.14.212.195:42281): Couldn't send request to peer d4e4a5c2ba664cd08c3aba1bb348ed75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b477b96739354b6ab43bf4ee230f4559. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:01.054980 19313 ts_tablet_manager.cc:902] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75: Initiating tablet copy from peer 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194:34515)
I20240502 01:53:01.056754 19313 tablet_copy_client.cc:308] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:34515
I20240502 01:53:01.063884 17383 tablet_copy_service.cc:140] P 4d190acabedb48b3bbb8cd6b677622da: Received BeginTabletCopySession request for tablet b477b96739354b6ab43bf4ee230f4559 from peer d4e4a5c2ba664cd08c3aba1bb348ed75 ({username='slave'} at 127.14.212.195:58475)
I20240502 01:53:01.064103 17383 tablet_copy_service.cc:161] P 4d190acabedb48b3bbb8cd6b677622da: Beginning new tablet copy session on tablet b477b96739354b6ab43bf4ee230f4559 from peer d4e4a5c2ba664cd08c3aba1bb348ed75 at {username='slave'} at 127.14.212.195:58475: session id = d4e4a5c2ba664cd08c3aba1bb348ed75-b477b96739354b6ab43bf4ee230f4559
I20240502 01:53:01.066298 17383 tablet_copy_source_session.cc:205] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:01.068919 19313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b477b96739354b6ab43bf4ee230f4559. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:01.075626 19313 tablet_copy_client.cc:783] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:01.075938 19313 tablet_copy_client.cc:647] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:01.078356 19313 tablet_copy_client.cc:523] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:01.081236 19313 tablet_bootstrap.cc:492] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75: Bootstrap starting.
I20240502 01:53:01.115876 19313 tablet_bootstrap.cc:492] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:01.116448 19313 tablet_bootstrap.cc:492] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75: Bootstrap complete.
I20240502 01:53:01.116729 19313 ts_tablet_manager.cc:1367] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75: Time spent bootstrapping tablet: real 0.036s user 0.028s sys 0.004s
I20240502 01:53:01.117671 19313 raft_consensus.cc:348] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:53:01.118185 19313 raft_consensus.cc:729] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e4a5c2ba664cd08c3aba1bb348ed75, State: Initialized, Role: FOLLOWER
I20240502 01:53:01.118415 19313 consensus_queue.cc:260] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } }
I20240502 01:53:01.119201 19313 ts_tablet_manager.cc:1398] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75: Time spent starting tablet: real 0.002s user 0.002s sys 0.004s
I20240502 01:53:01.120307 17383 tablet_copy_service.cc:342] P 4d190acabedb48b3bbb8cd6b677622da: Request end of tablet copy session d4e4a5c2ba664cd08c3aba1bb348ed75-b477b96739354b6ab43bf4ee230f4559 received from {username='slave'} at 127.14.212.195:58475
I20240502 01:53:01.120465 17383 tablet_copy_service.cc:433] P 4d190acabedb48b3bbb8cd6b677622da: ending tablet copy session d4e4a5c2ba664cd08c3aba1bb348ed75-b477b96739354b6ab43bf4ee230f4559 on tablet b477b96739354b6ab43bf4ee230f4559 with peer d4e4a5c2ba664cd08c3aba1bb348ed75
I20240502 01:53:01.625437 18477 raft_consensus.cc:1203] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:01.627424 19323 raft_consensus.cc:2932] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4e4a5c2ba664cd08c3aba1bb348ed75 (127.14.212.195) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } } }
I20240502 01:53:01.628566 18477 raft_consensus.cc:2932] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4e4a5c2ba664cd08c3aba1bb348ed75 (127.14.212.195) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } } }
I20240502 01:53:01.631281 19323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:36006 (request call id 0) took 1191 ms. Trace:
I20240502 01:53:01.631498 19323 rpcz_store.cc:276] 0502 01:53:00.439537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 01:53:00.439616 (+ 79us) service_pool.cc:225] Handling call
0502 01:53:00.440864 (+ 1248us) log.cc:842] Serialized 263 byte log entry
0502 01:53:01.631215 (+1190351us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":66,"raft.queue_time_us":370,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":267,"spinlock_wait_cycles":140544,"thread_start_us":518,"threads_started":2,"wal-append.queue_time_us":335,"wal-append.run_cpu_time_us":529,"wal-append.run_wall_time_us":1000596}
I20240502 01:53:01.633745 17078 catalog_manager.cc:5484] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75 reported cstate change: config changed from index -1 to 4, VOTER d4e4a5c2ba664cd08c3aba1bb348ed75 (127.14.212.195) added. New cstate: current_term: 1 leader_uuid: "4d190acabedb48b3bbb8cd6b677622da" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } } }
I20240502 01:53:01.679142 17230 consensus_queue.cc:237] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } }
W20240502 01:53:01.681432 17163 consensus_peers.cc:485] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 -> Peer d21ca31c11b746d99a102d116cfad615 (127.14.212.196:43697): Couldn't send request to peer d21ca31c11b746d99a102d116cfad615. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d94d0492ef1d4603b77441d09ad3c6dc. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:02.218238 19349 ts_tablet_manager.cc:902] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615: Initiating tablet copy from peer b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193:44487)
I20240502 01:53:02.219481 19349 tablet_copy_client.cc:308] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.193:44487
I20240502 01:53:02.220288 17250 tablet_copy_service.cc:140] P b8ffc52e71f045e5ac0f8aa35f3eea38: Received BeginTabletCopySession request for tablet d94d0492ef1d4603b77441d09ad3c6dc from peer d21ca31c11b746d99a102d116cfad615 ({username='slave'} at 127.14.212.196:53179)
I20240502 01:53:02.220520 17250 tablet_copy_service.cc:161] P b8ffc52e71f045e5ac0f8aa35f3eea38: Beginning new tablet copy session on tablet d94d0492ef1d4603b77441d09ad3c6dc from peer d21ca31c11b746d99a102d116cfad615 at {username='slave'} at 127.14.212.196:53179: session id = d21ca31c11b746d99a102d116cfad615-d94d0492ef1d4603b77441d09ad3c6dc
I20240502 01:53:02.222767 17250 tablet_copy_source_session.cc:205] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:02.224099 19349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94d0492ef1d4603b77441d09ad3c6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:02.229105 19349 tablet_copy_client.cc:783] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:02.229434 19349 tablet_copy_client.cc:647] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:02.231360 19349 tablet_copy_client.cc:523] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:02.233882 19349 tablet_bootstrap.cc:492] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615: Bootstrap starting.
I20240502 01:53:02.264566 19349 tablet_bootstrap.cc:492] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:02.265067 19349 tablet_bootstrap.cc:492] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615: Bootstrap complete.
I20240502 01:53:02.265347 19349 ts_tablet_manager.cc:1367] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.015s
I20240502 01:53:02.266222 19349 raft_consensus.cc:348] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:53:02.266615 19349 raft_consensus.cc:729] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d21ca31c11b746d99a102d116cfad615, State: Initialized, Role: FOLLOWER
I20240502 01:53:02.266808 19349 consensus_queue.cc:260] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } }
I20240502 01:53:02.267645 19349 ts_tablet_manager.cc:1398] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:53:02.268767 17250 tablet_copy_service.cc:342] P b8ffc52e71f045e5ac0f8aa35f3eea38: Request end of tablet copy session d21ca31c11b746d99a102d116cfad615-d94d0492ef1d4603b77441d09ad3c6dc received from {username='slave'} at 127.14.212.196:53179
I20240502 01:53:02.268923 17250 tablet_copy_service.cc:433] P b8ffc52e71f045e5ac0f8aa35f3eea38: ending tablet copy session d21ca31c11b746d99a102d116cfad615-d94d0492ef1d4603b77441d09ad3c6dc on tablet d94d0492ef1d4603b77441d09ad3c6dc with peer d21ca31c11b746d99a102d116cfad615
I20240502 01:53:02.765965 18619 raft_consensus.cc:1203] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:02.767416 19356 raft_consensus.cc:2932] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d21ca31c11b746d99a102d116cfad615 (127.14.212.196) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } } }
I20240502 01:53:02.768168 18619 raft_consensus.cc:2932] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d21ca31c11b746d99a102d116cfad615 (127.14.212.196) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } } }
I20240502 01:53:02.771380 19356 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:34134 (request call id 0) took 1092 ms. Trace:
I20240502 01:53:02.771721 19356 rpcz_store.cc:276] 0502 01:53:01.678536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 01:53:01.678629 (+ 93us) service_pool.cc:225] Handling call
0502 01:53:01.679759 (+ 1130us) log.cc:842] Serialized 263 byte log entry
0502 01:53:02.771315 (+1091556us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":73,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":228,"thread_start_us":214,"threads_started":1,"wal-append.queue_time_us":315,"wal-append.run_cpu_time_us":517,"wal-append.run_wall_time_us":1000596}
I20240502 01:53:02.773866 17078 catalog_manager.cc:5484] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615 reported cstate change: config changed from index -1 to 4, VOTER d21ca31c11b746d99a102d116cfad615 (127.14.212.196) added. New cstate: current_term: 1 leader_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } } }
I20240502 01:53:02.813143 17362 consensus_queue.cc:237] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } }
W20240502 01:53:02.815340 17298 consensus_peers.cc:485] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da -> Peer adf51a82733d4010b3028cb54cc307e2 (127.14.212.197:41667): Couldn't send request to peer adf51a82733d4010b3028cb54cc307e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97ffbdb149fb42c892ed094ce60f7166. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:03.331310 19382 ts_tablet_manager.cc:902] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2: Initiating tablet copy from peer 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194:34515)
I20240502 01:53:03.332823 19382 tablet_copy_client.cc:308] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:34515
I20240502 01:53:03.333662 17383 tablet_copy_service.cc:140] P 4d190acabedb48b3bbb8cd6b677622da: Received BeginTabletCopySession request for tablet 97ffbdb149fb42c892ed094ce60f7166 from peer adf51a82733d4010b3028cb54cc307e2 ({username='slave'} at 127.14.212.197:41789)
I20240502 01:53:03.333818 17383 tablet_copy_service.cc:161] P 4d190acabedb48b3bbb8cd6b677622da: Beginning new tablet copy session on tablet 97ffbdb149fb42c892ed094ce60f7166 from peer adf51a82733d4010b3028cb54cc307e2 at {username='slave'} at 127.14.212.197:41789: session id = adf51a82733d4010b3028cb54cc307e2-97ffbdb149fb42c892ed094ce60f7166
I20240502 01:53:03.335917 17383 tablet_copy_source_session.cc:205] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:03.337201 19382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ffbdb149fb42c892ed094ce60f7166. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:03.341879 19382 tablet_copy_client.cc:783] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:03.342192 19382 tablet_copy_client.cc:647] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:03.344014 19382 tablet_copy_client.cc:523] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:03.346755 19382 tablet_bootstrap.cc:492] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2: Bootstrap starting.
I20240502 01:53:03.376770 19382 tablet_bootstrap.cc:492] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:03.377265 19382 tablet_bootstrap.cc:492] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2: Bootstrap complete.
I20240502 01:53:03.377534 19382 ts_tablet_manager.cc:1367] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2: Time spent bootstrapping tablet: real 0.031s user 0.028s sys 0.003s
I20240502 01:53:03.378401 19382 raft_consensus.cc:348] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:53:03.378846 19382 raft_consensus.cc:729] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf51a82733d4010b3028cb54cc307e2, State: Initialized, Role: FOLLOWER
I20240502 01:53:03.379033 19382 consensus_queue.cc:260] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } }
I20240502 01:53:03.379988 19382 ts_tablet_manager.cc:1398] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:03.381058 17383 tablet_copy_service.cc:342] P 4d190acabedb48b3bbb8cd6b677622da: Request end of tablet copy session adf51a82733d4010b3028cb54cc307e2-97ffbdb149fb42c892ed094ce60f7166 received from {username='slave'} at 127.14.212.197:41789
I20240502 01:53:03.381258 17383 tablet_copy_service.cc:433] P 4d190acabedb48b3bbb8cd6b677622da: ending tablet copy session adf51a82733d4010b3028cb54cc307e2-97ffbdb149fb42c892ed094ce60f7166 on tablet 97ffbdb149fb42c892ed094ce60f7166 with peer adf51a82733d4010b3028cb54cc307e2
I20240502 01:53:03.725147 18751 raft_consensus.cc:1203] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:03.726795 19389 raft_consensus.cc:2932] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER adf51a82733d4010b3028cb54cc307e2 (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } } }
I20240502 01:53:03.727459 18751 raft_consensus.cc:2932] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER adf51a82733d4010b3028cb54cc307e2 (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } } }
I20240502 01:53:03.732790 17080 catalog_manager.cc:5484] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da reported cstate change: config changed from index -1 to 4, VOTER adf51a82733d4010b3028cb54cc307e2 (127.14.212.197) added. New cstate: current_term: 1 leader_uuid: "4d190acabedb48b3bbb8cd6b677622da" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } } }
I20240502 01:53:03.767911 17362 consensus_queue.cc:237] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } }
W20240502 01:53:03.777333 17295 consensus_peers.cc:485] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da -> Peer aaf6d9dc6f5747bd83d2b9e24a7d679a (127.14.212.198:34487): Couldn't send request to peer aaf6d9dc6f5747bd83d2b9e24a7d679a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1636e2cd56f40359df64d2ac4c9d806. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:04.365073 19414 ts_tablet_manager.cc:902] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a: Initiating tablet copy from peer 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194:34515)
I20240502 01:53:04.366365 19414 tablet_copy_client.cc:308] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:34515
I20240502 01:53:04.373251 17383 tablet_copy_service.cc:140] P 4d190acabedb48b3bbb8cd6b677622da: Received BeginTabletCopySession request for tablet b1636e2cd56f40359df64d2ac4c9d806 from peer aaf6d9dc6f5747bd83d2b9e24a7d679a ({username='slave'} at 127.14.212.198:59605)
I20240502 01:53:04.373451 17383 tablet_copy_service.cc:161] P 4d190acabedb48b3bbb8cd6b677622da: Beginning new tablet copy session on tablet b1636e2cd56f40359df64d2ac4c9d806 from peer aaf6d9dc6f5747bd83d2b9e24a7d679a at {username='slave'} at 127.14.212.198:59605: session id = aaf6d9dc6f5747bd83d2b9e24a7d679a-b1636e2cd56f40359df64d2ac4c9d806
I20240502 01:53:04.375538 17383 tablet_copy_source_session.cc:205] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:04.376829 19414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1636e2cd56f40359df64d2ac4c9d806. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:04.381536 19414 tablet_copy_client.cc:783] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:04.381863 19414 tablet_copy_client.cc:647] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:04.383805 19414 tablet_copy_client.cc:523] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:04.386272 19414 tablet_bootstrap.cc:492] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a: Bootstrap starting.
I20240502 01:53:04.415845 19414 tablet_bootstrap.cc:492] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:04.416384 19414 tablet_bootstrap.cc:492] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a: Bootstrap complete.
I20240502 01:53:04.416656 19414 ts_tablet_manager.cc:1367] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.004s
I20240502 01:53:04.417595 19414 raft_consensus.cc:348] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:53:04.417986 19414 raft_consensus.cc:729] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf6d9dc6f5747bd83d2b9e24a7d679a, State: Initialized, Role: FOLLOWER
I20240502 01:53:04.418167 19414 consensus_queue.cc:260] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } }
I20240502 01:53:04.418931 19414 ts_tablet_manager.cc:1398] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 01:53:04.419970 17383 tablet_copy_service.cc:342] P 4d190acabedb48b3bbb8cd6b677622da: Request end of tablet copy session aaf6d9dc6f5747bd83d2b9e24a7d679a-b1636e2cd56f40359df64d2ac4c9d806 received from {username='slave'} at 127.14.212.198:59605
I20240502 01:53:04.420140 17383 tablet_copy_service.cc:433] P 4d190acabedb48b3bbb8cd6b677622da: ending tablet copy session aaf6d9dc6f5747bd83d2b9e24a7d679a-b1636e2cd56f40359df64d2ac4c9d806 on tablet b1636e2cd56f40359df64d2ac4c9d806 with peer aaf6d9dc6f5747bd83d2b9e24a7d679a
I20240502 01:53:04.834465 18884 raft_consensus.cc:1203] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:04.835988 19423 raft_consensus.cc:2932] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aaf6d9dc6f5747bd83d2b9e24a7d679a (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } } }
I20240502 01:53:04.836642 18884 raft_consensus.cc:2932] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aaf6d9dc6f5747bd83d2b9e24a7d679a (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } } }
I20240502 01:53:04.839183 19423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:36044 (request call id 0) took 1071 ms. Trace:
I20240502 01:53:04.839371 19423 rpcz_store.cc:276] 0502 01:53:03.767314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 01:53:03.767412 (+ 98us) service_pool.cc:225] Handling call
0502 01:53:03.768677 (+ 1265us) log.cc:842] Serialized 263 byte log entry
0502 01:53:04.839123 (+1070446us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":94,"raft.queue_time_us":52,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":258,"spinlock_wait_cycles":33664,"thread_start_us":334,"threads_started":1,"wal-append.queue_time_us":461,"wal-append.run_cpu_time_us":642,"wal-append.run_wall_time_us":1000705}
I20240502 01:53:04.841984 17078 catalog_manager.cc:5484] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a reported cstate change: config changed from index -1 to 4, VOTER aaf6d9dc6f5747bd83d2b9e24a7d679a (127.14.212.198) added. New cstate: current_term: 1 leader_uuid: "4d190acabedb48b3bbb8cd6b677622da" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } } }
I20240502 01:53:04.879535 17230 consensus_queue.cc:237] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } }
W20240502 01:53:04.881827 17163 consensus_peers.cc:485] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 -> Peer f7baee4920524734aa33ce500a559f3b (127.14.212.199:45747): Couldn't send request to peer f7baee4920524734aa33ce500a559f3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 037045b61419472089771a8c6a057d21. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:05.458655 19448 ts_tablet_manager.cc:902] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b: Initiating tablet copy from peer b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193:44487)
I20240502 01:53:05.460129 19448 tablet_copy_client.cc:308] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.193:44487
I20240502 01:53:05.460974 17250 tablet_copy_service.cc:140] P b8ffc52e71f045e5ac0f8aa35f3eea38: Received BeginTabletCopySession request for tablet 037045b61419472089771a8c6a057d21 from peer f7baee4920524734aa33ce500a559f3b ({username='slave'} at 127.14.212.199:38983)
I20240502 01:53:05.461138 17250 tablet_copy_service.cc:161] P b8ffc52e71f045e5ac0f8aa35f3eea38: Beginning new tablet copy session on tablet 037045b61419472089771a8c6a057d21 from peer f7baee4920524734aa33ce500a559f3b at {username='slave'} at 127.14.212.199:38983: session id = f7baee4920524734aa33ce500a559f3b-037045b61419472089771a8c6a057d21
I20240502 01:53:05.463325 17250 tablet_copy_source_session.cc:205] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:05.464690 19448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 037045b61419472089771a8c6a057d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:05.469485 19448 tablet_copy_client.cc:783] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:05.469803 19448 tablet_copy_client.cc:647] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:05.471730 19448 tablet_copy_client.cc:523] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:05.474354 19448 tablet_bootstrap.cc:492] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b: Bootstrap starting.
I20240502 01:53:05.505293 19448 tablet_bootstrap.cc:492] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:05.505748 19448 tablet_bootstrap.cc:492] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b: Bootstrap complete.
I20240502 01:53:05.506013 19448 ts_tablet_manager.cc:1367] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b: Time spent bootstrapping tablet: real 0.032s user 0.028s sys 0.004s
I20240502 01:53:05.506886 19448 raft_consensus.cc:348] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:53:05.507254 19448 raft_consensus.cc:729] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7baee4920524734aa33ce500a559f3b, State: Initialized, Role: FOLLOWER
I20240502 01:53:05.507414 19448 consensus_queue.cc:260] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } }
I20240502 01:53:05.508162 19448 ts_tablet_manager.cc:1398] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:05.509130 17250 tablet_copy_service.cc:342] P b8ffc52e71f045e5ac0f8aa35f3eea38: Request end of tablet copy session f7baee4920524734aa33ce500a559f3b-037045b61419472089771a8c6a057d21 received from {username='slave'} at 127.14.212.199:38983
I20240502 01:53:05.509316 17250 tablet_copy_service.cc:433] P b8ffc52e71f045e5ac0f8aa35f3eea38: ending tablet copy session f7baee4920524734aa33ce500a559f3b-037045b61419472089771a8c6a057d21 on tablet 037045b61419472089771a8c6a057d21 with peer f7baee4920524734aa33ce500a559f3b
I20240502 01:53:05.996531 19016 raft_consensus.cc:1203] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:05.997807 19456 raft_consensus.cc:2932] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f7baee4920524734aa33ce500a559f3b (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } } }
I20240502 01:53:05.998493 19016 raft_consensus.cc:2932] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f7baee4920524734aa33ce500a559f3b (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } } }
I20240502 01:53:06.000958 19456 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:34156 (request call id 0) took 1121 ms. Trace:
I20240502 01:53:06.001205 19456 rpcz_store.cc:276] 0502 01:53:04.878988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 01:53:04.879058 (+ 70us) service_pool.cc:225] Handling call
0502 01:53:04.880229 (+ 1171us) log.cc:842] Serialized 263 byte log entry
0502 01:53:06.000901 (+1120672us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":53,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":189,"spinlock_wait_cycles":29952,"thread_start_us":240,"threads_started":1,"wal-append.queue_time_us":350,"wal-append.run_cpu_time_us":615,"wal-append.run_wall_time_us":1000693}
I20240502 01:53:06.003304 17080 catalog_manager.cc:5484] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b reported cstate change: config changed from index -1 to 4, VOTER f7baee4920524734aa33ce500a559f3b (127.14.212.199) added. New cstate: current_term: 1 leader_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } } }
I20240502 01:53:06.145532 18730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.162185 18598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.164372 18455 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:06.176446 18996 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.177244 18864 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.192713 17342 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:53:06.205507 17209 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:53:06.279579 17230 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "037045b61419472089771a8c6a057d21"
dest_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34166
I20240502 01:53:06.279894 17230 raft_consensus.cc:595] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:06.382594 17342 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:53:06.389925 17209 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:53:06.410264 18864 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.419430 19456 raft_consensus.cc:979] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38: : Instructing follower f7baee4920524734aa33ce500a559f3b to start an election
I20240502 01:53:06.419643 19456 raft_consensus.cc:1067] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Signalling peer f7baee4920524734aa33ce500a559f3b to start an election
I20240502 01:53:06.421510 19016 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "037045b61419472089771a8c6a057d21"
dest_uuid: "f7baee4920524734aa33ce500a559f3b"
from {username='slave'} at 127.14.212.193:60289
I20240502 01:53:06.421809 19016 raft_consensus.cc:482] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:06.421988 19016 raft_consensus.cc:3037] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:06.422705 18730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.428555 18598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.432832 19016 raft_consensus.cc:504] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } }
I20240502 01:53:06.441785 18996 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.444665 17230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037045b61419472089771a8c6a057d21" candidate_uuid: "f7baee4920524734aa33ce500a559f3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38"
I20240502 01:53:06.445111 17230 raft_consensus.cc:3032] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:06.445212 17230 raft_consensus.cc:729] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Running, Role: LEADER
I20240502 01:53:06.445587 17230 consensus_queue.cc:260] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } }
I20240502 01:53:06.445838 17230 raft_consensus.cc:3037] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:06.449443 17230 raft_consensus.cc:2454] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7baee4920524734aa33ce500a559f3b in term 2.
I20240502 01:53:06.450134 18950 leader_election.cc:302] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b8ffc52e71f045e5ac0f8aa35f3eea38, f7baee4920524734aa33ce500a559f3b; no voters:
I20240502 01:53:06.450660 19454 raft_consensus.cc:2781] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:06.451350 19454 raft_consensus.cc:686] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b [term 2 LEADER]: Becoming Leader. State: Replica: f7baee4920524734aa33ce500a559f3b, State: Running, Role: LEADER
I20240502 01:53:06.451961 19454 consensus_queue.cc:237] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } }
I20240502 01:53:06.458819 17078 catalog_manager.cc:5484] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b reported cstate change: term changed from 1 to 2, leader changed from b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193) to f7baee4920524734aa33ce500a559f3b (127.14.212.199). New cstate: current_term: 2 leader_uuid: "f7baee4920524734aa33ce500a559f3b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } } }
I20240502 01:53:06.461091 19016 leader_election.cc:288] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b [CANDIDATE]: Term 2 election: Requested vote from peers b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193:44487)
I20240502 01:53:06.484597 18455 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:06.649124 17342 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:53:06.656144 18864 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.680351 18730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.696110 18598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.716053 17209 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:53:06.726047 18455 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:06.729707 18996 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:06.804450 17362 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1636e2cd56f40359df64d2ac4c9d806"
dest_uuid: "4d190acabedb48b3bbb8cd6b677622da"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36098
I20240502 01:53:06.804731 17362 raft_consensus.cc:595] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:06.914108 17342 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:53:06.925910 18864 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.936810 18996 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:06.937315 18598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.938350 18730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.949647 17209 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:53:06.954784 18455 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:07.037954 19622 raft_consensus.cc:979] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da: : Instructing follower aaf6d9dc6f5747bd83d2b9e24a7d679a to start an election
I20240502 01:53:07.038136 19622 raft_consensus.cc:1067] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Signalling peer aaf6d9dc6f5747bd83d2b9e24a7d679a to start an election
I20240502 01:53:07.039324 18884 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b1636e2cd56f40359df64d2ac4c9d806"
dest_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a"
from {username='slave'} at 127.14.212.194:36759
I20240502 01:53:07.039645 18884 raft_consensus.cc:482] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:07.039851 18884 raft_consensus.cc:3037] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:07.042284 18884 raft_consensus.cc:504] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } }
I20240502 01:53:07.043943 18884 leader_election.cc:288] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a [CANDIDATE]: Term 2 election: Requested vote from peers 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194:34515)
I20240502 01:53:07.044622 17362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1636e2cd56f40359df64d2ac4c9d806" candidate_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4d190acabedb48b3bbb8cd6b677622da"
I20240502 01:53:07.044982 17362 raft_consensus.cc:3032] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:07.045100 17362 raft_consensus.cc:729] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d190acabedb48b3bbb8cd6b677622da, State: Running, Role: LEADER
I20240502 01:53:07.045447 17362 consensus_queue.cc:260] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } }
I20240502 01:53:07.045902 17362 raft_consensus.cc:3037] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:07.047906 17362 raft_consensus.cc:2454] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf6d9dc6f5747bd83d2b9e24a7d679a in term 2.
I20240502 01:53:07.048492 18815 leader_election.cc:302] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d190acabedb48b3bbb8cd6b677622da, aaf6d9dc6f5747bd83d2b9e24a7d679a; no voters:
I20240502 01:53:07.049507 19628 raft_consensus.cc:2781] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:07.050055 19628 raft_consensus.cc:686] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 2 LEADER]: Becoming Leader. State: Replica: aaf6d9dc6f5747bd83d2b9e24a7d679a, State: Running, Role: LEADER
I20240502 01:53:07.050637 19628 consensus_queue.cc:237] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } }
I20240502 01:53:07.054879 17078 catalog_manager.cc:5484] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a reported cstate change: term changed from 1 to 2, leader changed from 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194) to aaf6d9dc6f5747bd83d2b9e24a7d679a (127.14.212.198). New cstate: current_term: 2 leader_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } } }
I20240502 01:53:07.057986 17230 raft_consensus.cc:1261] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Refusing update from remote peer f7baee4920524734aa33ce500a559f3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:53:07.059079 19637 consensus_queue.cc:1035] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:07.161360 17342 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:53:07.182777 18864 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:07.187521 18730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.188387 18598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.192138 17209 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:53:07.205940 18455 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:07.220163 18996 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:07.309944 17361 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3fed2519c9b34871b27e5148e18b0673"
dest_uuid: "4d190acabedb48b3bbb8cd6b677622da"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36134
I20240502 01:53:07.310186 17361 raft_consensus.cc:595] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:07.407353 17342 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:53:07.410890 18864 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:07.415978 18730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.422727 18598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.436924 18996 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:07.441041 18455 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:07.457073 17209 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:53:07.502640 17361 raft_consensus.cc:1261] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Refusing update from remote peer aaf6d9dc6f5747bd83d2b9e24a7d679a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:53:07.503798 19628 consensus_queue.cc:1035] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:07.630858 18864 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:07.654721 18598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.656144 17209 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:53:07.674868 17342 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:53:07.687093 18730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.690093 18996 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:07.699666 18455 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:07.789417 17361 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "97ffbdb149fb42c892ed094ce60f7166"
dest_uuid: "4d190acabedb48b3bbb8cd6b677622da"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36158
I20240502 01:53:07.789687 17361 raft_consensus.cc:595] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:07.792896 19758 raft_consensus.cc:979] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da: : Instructing follower adf51a82733d4010b3028cb54cc307e2 to start an election
I20240502 01:53:07.793161 19758 raft_consensus.cc:1067] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Signalling peer adf51a82733d4010b3028cb54cc307e2 to start an election
I20240502 01:53:07.794181 18751 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3fed2519c9b34871b27e5148e18b0673"
dest_uuid: "adf51a82733d4010b3028cb54cc307e2"
from {username='slave'} at 127.14.212.194:49551
I20240502 01:53:07.794495 18751 raft_consensus.cc:482] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:07.794723 18751 raft_consensus.cc:3037] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:07.797780 18751 raft_consensus.cc:504] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } }
I20240502 01:53:07.799392 18751 leader_election.cc:288] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2 [CANDIDATE]: Term 2 election: Requested vote from peers 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194:34515)
I20240502 01:53:07.800148 17361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fed2519c9b34871b27e5148e18b0673" candidate_uuid: "adf51a82733d4010b3028cb54cc307e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4d190acabedb48b3bbb8cd6b677622da"
I20240502 01:53:07.800388 17361 raft_consensus.cc:3032] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:07.800458 17361 raft_consensus.cc:729] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d190acabedb48b3bbb8cd6b677622da, State: Running, Role: LEADER
I20240502 01:53:07.800756 17361 consensus_queue.cc:260] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } }
I20240502 01:53:07.801082 17361 raft_consensus.cc:3037] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:07.804497 17361 raft_consensus.cc:2454] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adf51a82733d4010b3028cb54cc307e2 in term 2.
I20240502 01:53:07.805321 18683 leader_election.cc:302] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d190acabedb48b3bbb8cd6b677622da, adf51a82733d4010b3028cb54cc307e2; no voters:
I20240502 01:53:07.806157 19764 raft_consensus.cc:2781] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:07.806682 19764 raft_consensus.cc:686] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2 [term 2 LEADER]: Becoming Leader. State: Replica: adf51a82733d4010b3028cb54cc307e2, State: Running, Role: LEADER
I20240502 01:53:07.808732 19764 consensus_queue.cc:237] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } }
I20240502 01:53:07.813277 17084 catalog_manager.cc:5484] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2 reported cstate change: term changed from 1 to 2, leader changed from 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194) to adf51a82733d4010b3028cb54cc307e2 (127.14.212.197). New cstate: current_term: 2 leader_uuid: "adf51a82733d4010b3028cb54cc307e2" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } } }
I20240502 01:53:07.882846 19622 raft_consensus.cc:979] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da: : Instructing follower adf51a82733d4010b3028cb54cc307e2 to start an election
I20240502 01:53:07.883041 19622 raft_consensus.cc:1067] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Signalling peer adf51a82733d4010b3028cb54cc307e2 to start an election
I20240502 01:53:07.883755 18751 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "97ffbdb149fb42c892ed094ce60f7166"
dest_uuid: "adf51a82733d4010b3028cb54cc307e2"
from {username='slave'} at 127.14.212.194:49551
I20240502 01:53:07.884047 18751 raft_consensus.cc:482] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:07.884179 18751 raft_consensus.cc:3037] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:07.887475 18751 raft_consensus.cc:504] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } }
I20240502 01:53:07.888285 18751 leader_election.cc:288] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2 [CANDIDATE]: Term 2 election: Requested vote from peers 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194:34515)
I20240502 01:53:07.890547 17361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ffbdb149fb42c892ed094ce60f7166" candidate_uuid: "adf51a82733d4010b3028cb54cc307e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4d190acabedb48b3bbb8cd6b677622da"
I20240502 01:53:07.893057 17361 raft_consensus.cc:3032] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:07.893215 17361 raft_consensus.cc:729] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d190acabedb48b3bbb8cd6b677622da, State: Running, Role: LEADER
I20240502 01:53:07.893620 17361 consensus_queue.cc:260] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } }
I20240502 01:53:07.894095 17361 raft_consensus.cc:3037] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:07.898583 17361 raft_consensus.cc:2454] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adf51a82733d4010b3028cb54cc307e2 in term 2.
I20240502 01:53:07.899194 18683 leader_election.cc:302] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d190acabedb48b3bbb8cd6b677622da, adf51a82733d4010b3028cb54cc307e2; no voters:
I20240502 01:53:07.901175 19764 raft_consensus.cc:2781] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:07.901399 19764 raft_consensus.cc:686] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2 [term 2 LEADER]: Becoming Leader. State: Replica: adf51a82733d4010b3028cb54cc307e2, State: Running, Role: LEADER
I20240502 01:53:07.901751 19764 consensus_queue.cc:237] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } }
I20240502 01:53:07.908223 17342 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:07.913492 17084 catalog_manager.cc:5484] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2 reported cstate change: term changed from 1 to 2, leader changed from 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194) to adf51a82733d4010b3028cb54cc307e2 (127.14.212.197). New cstate: current_term: 2 leader_uuid: "adf51a82733d4010b3028cb54cc307e2" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } } }
I20240502 01:53:07.928092 18864 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:07.929543 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:07.929761 18598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.932209 17209 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:53:07.954375 18996 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:07.959618 18455 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:08.133713 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:08.137804 17342 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:08.142356 18598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:08.143203 18455 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:08.143340 18864 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:08.156895 18996 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:08.172946 17209 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:53:08.254654 17361 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b477b96739354b6ab43bf4ee230f4559"
dest_uuid: "4d190acabedb48b3bbb8cd6b677622da"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36184
I20240502 01:53:08.254946 17361 raft_consensus.cc:595] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:08.285621 17361 raft_consensus.cc:1261] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Refusing update from remote peer adf51a82733d4010b3028cb54cc307e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:53:08.286466 19764 consensus_queue.cc:1035] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:08.371057 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:08.383033 18864 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:08.391091 17342 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:08.400599 18455 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:08.408593 18996 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:08.409720 17209 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:53:08.426082 18598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:08.490501 17361 raft_consensus.cc:1261] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Refusing update from remote peer adf51a82733d4010b3028cb54cc307e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:53:08.491255 19765 consensus_queue.cc:1035] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:08.513568 19622 raft_consensus.cc:979] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da: : Instructing follower d4e4a5c2ba664cd08c3aba1bb348ed75 to start an election
I20240502 01:53:08.513757 19622 raft_consensus.cc:1067] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Signalling peer d4e4a5c2ba664cd08c3aba1bb348ed75 to start an election
I20240502 01:53:08.514621 18477 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b477b96739354b6ab43bf4ee230f4559"
dest_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75"
from {username='slave'} at 127.14.212.194:33473
I20240502 01:53:08.514964 18477 raft_consensus.cc:482] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:08.515112 18477 raft_consensus.cc:3037] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:08.518241 18477 raft_consensus.cc:504] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } }
I20240502 01:53:08.519670 18477 leader_election.cc:288] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75 [CANDIDATE]: Term 2 election: Requested vote from peers 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194:34515)
I20240502 01:53:08.520406 17361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b477b96739354b6ab43bf4ee230f4559" candidate_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4d190acabedb48b3bbb8cd6b677622da"
I20240502 01:53:08.520645 17361 raft_consensus.cc:3032] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:08.520733 17361 raft_consensus.cc:729] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d190acabedb48b3bbb8cd6b677622da, State: Running, Role: LEADER
I20240502 01:53:08.521057 17361 consensus_queue.cc:260] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } }
I20240502 01:53:08.521376 17361 raft_consensus.cc:3037] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:08.523667 17361 raft_consensus.cc:2454] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4e4a5c2ba664cd08c3aba1bb348ed75 in term 2.
I20240502 01:53:08.524343 18406 leader_election.cc:302] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d190acabedb48b3bbb8cd6b677622da, d4e4a5c2ba664cd08c3aba1bb348ed75; no voters:
I20240502 01:53:08.525101 19891 raft_consensus.cc:2781] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:08.525563 19891 raft_consensus.cc:686] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 2 LEADER]: Becoming Leader. State: Replica: d4e4a5c2ba664cd08c3aba1bb348ed75, State: Running, Role: LEADER
I20240502 01:53:08.525982 19891 consensus_queue.cc:237] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } }
I20240502 01:53:08.529191 17078 catalog_manager.cc:5484] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75 reported cstate change: term changed from 1 to 2, leader changed from 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194) to d4e4a5c2ba664cd08c3aba1bb348ed75 (127.14.212.195). New cstate: current_term: 2 leader_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } } }
I20240502 01:53:08.610853 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:08.629125 18864 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:08.642899 17209 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:53:08.650401 18598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:08.663736 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:08.697378 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:08.697937 18996 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:08.785900 17230 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bc1d0f4745df4203848a1f4820f9b499"
dest_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59416
I20240502 01:53:08.786116 17230 raft_consensus.cc:595] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:08.889331 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:08.893421 18864 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:08.904845 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:08.933574 17209 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:53:08.935940 18598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:08.938302 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:08.972690 18996 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:09.125948 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:09.134752 18864 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:09.146582 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:09.151371 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:09.181602 18598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:09.182933 17209 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:53:09.205240 17361 raft_consensus.cc:1261] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Refusing update from remote peer d4e4a5c2ba664cd08c3aba1bb348ed75: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:53:09.207031 19998 consensus_queue.cc:1035] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:09.213755 18996 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:09.287433 20003 raft_consensus.cc:979] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38: : Instructing follower f7baee4920524734aa33ce500a559f3b to start an election
I20240502 01:53:09.287752 20003 raft_consensus.cc:1067] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Signalling peer f7baee4920524734aa33ce500a559f3b to start an election
I20240502 01:53:09.288904 19016 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bc1d0f4745df4203848a1f4820f9b499"
dest_uuid: "f7baee4920524734aa33ce500a559f3b"
from {username='slave'} at 127.14.212.193:60289
I20240502 01:53:09.289207 19016 raft_consensus.cc:482] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:09.289361 19016 raft_consensus.cc:3037] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:09.291960 19016 raft_consensus.cc:504] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } }
I20240502 01:53:09.292465 19016 leader_election.cc:288] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b [CANDIDATE]: Term 2 election: Requested vote from peers b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193:44487)
I20240502 01:53:09.293100 17230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1d0f4745df4203848a1f4820f9b499" candidate_uuid: "f7baee4920524734aa33ce500a559f3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38"
I20240502 01:53:09.293490 17230 raft_consensus.cc:3032] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:09.293641 17230 raft_consensus.cc:729] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Running, Role: LEADER
I20240502 01:53:09.293987 17230 consensus_queue.cc:260] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } }
I20240502 01:53:09.294432 17230 raft_consensus.cc:3037] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:09.296504 17230 raft_consensus.cc:2454] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7baee4920524734aa33ce500a559f3b in term 2.
I20240502 01:53:09.297187 18950 leader_election.cc:302] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b8ffc52e71f045e5ac0f8aa35f3eea38, f7baee4920524734aa33ce500a559f3b; no voters:
I20240502 01:53:09.297559 19841 raft_consensus.cc:2781] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:09.299788 19841 raft_consensus.cc:686] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b [term 2 LEADER]: Becoming Leader. State: Replica: f7baee4920524734aa33ce500a559f3b, State: Running, Role: LEADER
I20240502 01:53:09.300220 19841 consensus_queue.cc:237] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } }
I20240502 01:53:09.305838 17081 catalog_manager.cc:5484] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b reported cstate change: term changed from 1 to 2, leader changed from b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193) to f7baee4920524734aa33ce500a559f3b (127.14.212.199). New cstate: current_term: 2 leader_uuid: "f7baee4920524734aa33ce500a559f3b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } } }
I20240502 01:53:09.310451 17230 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e4a22d109d8044adaeb364037e9c137e"
dest_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59460
I20240502 01:53:09.310724 17230 raft_consensus.cc:595] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:09.432219 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:09.440505 18864 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:09.449340 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:09.457901 18598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:09.474895 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:09.495538 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:09.497632 17209 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:53:09.616063 19356 raft_consensus.cc:979] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38: : Instructing follower d21ca31c11b746d99a102d116cfad615 to start an election
I20240502 01:53:09.616271 19356 raft_consensus.cc:1067] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Signalling peer d21ca31c11b746d99a102d116cfad615 to start an election
I20240502 01:53:09.617146 18619 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e4a22d109d8044adaeb364037e9c137e"
dest_uuid: "d21ca31c11b746d99a102d116cfad615"
from {username='slave'} at 127.14.212.193:48237
I20240502 01:53:09.617506 18619 raft_consensus.cc:482] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:09.617762 18619 raft_consensus.cc:3037] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:09.621356 18619 raft_consensus.cc:504] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } }
I20240502 01:53:09.623167 18619 leader_election.cc:288] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615 [CANDIDATE]: Term 2 election: Requested vote from peers b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193:44487)
I20240502 01:53:09.624179 17230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a22d109d8044adaeb364037e9c137e" candidate_uuid: "d21ca31c11b746d99a102d116cfad615" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38"
I20240502 01:53:09.624444 17230 raft_consensus.cc:3032] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:09.624650 17230 raft_consensus.cc:729] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Running, Role: LEADER
I20240502 01:53:09.624976 17230 consensus_queue.cc:260] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } }
I20240502 01:53:09.625394 17230 raft_consensus.cc:3037] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:09.628521 17230 raft_consensus.cc:2454] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d21ca31c11b746d99a102d116cfad615 in term 2.
I20240502 01:53:09.630087 18552 leader_election.cc:302] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b8ffc52e71f045e5ac0f8aa35f3eea38, d21ca31c11b746d99a102d116cfad615; no voters:
I20240502 01:53:09.635560 20073 raft_consensus.cc:2781] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:09.637377 20073 raft_consensus.cc:686] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615 [term 2 LEADER]: Becoming Leader. State: Replica: d21ca31c11b746d99a102d116cfad615, State: Running, Role: LEADER
I20240502 01:53:09.638149 20073 consensus_queue.cc:237] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } }
I20240502 01:53:09.653381 17083 catalog_manager.cc:5484] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615 reported cstate change: term changed from 1 to 2, leader changed from b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193) to d21ca31c11b746d99a102d116cfad615 (127.14.212.196). New cstate: current_term: 2 leader_uuid: "d21ca31c11b746d99a102d116cfad615" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } } }
I20240502 01:53:09.678629 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:09.679219 17230 raft_consensus.cc:1261] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Refusing update from remote peer f7baee4920524734aa33ce500a559f3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:53:09.679977 19841 consensus_queue.cc:1035] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 01:53:09.693549 18864 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:09.695398 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:09.696091 17209 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:09.712538 18598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:09.721659 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:09.764073 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:09.823390 17230 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1e8092e428084a56bbfaaca963877631"
dest_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59478
I20240502 01:53:09.823650 17230 raft_consensus.cc:595] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:09.930157 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:09.935622 18864 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:09.946056 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:09.959750 17209 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:09.984977 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:09.985139 18598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:09.993433 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:10.092298 17230 raft_consensus.cc:1261] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Refusing update from remote peer d21ca31c11b746d99a102d116cfad615: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:53:10.093143 20073 consensus_queue.cc:1035] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:10.138561 19356 raft_consensus.cc:979] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38: : Instructing follower aaf6d9dc6f5747bd83d2b9e24a7d679a to start an election
I20240502 01:53:10.138764 19356 raft_consensus.cc:1067] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Signalling peer aaf6d9dc6f5747bd83d2b9e24a7d679a to start an election
I20240502 01:53:10.139405 18884 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e8092e428084a56bbfaaca963877631"
dest_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a"
from {username='slave'} at 127.14.212.193:34633
I20240502 01:53:10.139636 18884 raft_consensus.cc:482] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:10.139750 18884 raft_consensus.cc:3037] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:10.142354 18884 raft_consensus.cc:504] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } }
I20240502 01:53:10.144280 18884 leader_election.cc:288] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a [CANDIDATE]: Term 2 election: Requested vote from peers b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193:44487)
I20240502 01:53:10.146005 17230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8092e428084a56bbfaaca963877631" candidate_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38"
I20240502 01:53:10.146284 17230 raft_consensus.cc:3032] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:10.146394 17230 raft_consensus.cc:729] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Running, Role: LEADER
I20240502 01:53:10.146688 17230 consensus_queue.cc:260] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } }
I20240502 01:53:10.147049 17230 raft_consensus.cc:3037] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:10.149698 17230 raft_consensus.cc:2454] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf6d9dc6f5747bd83d2b9e24a7d679a in term 2.
I20240502 01:53:10.150296 18817 leader_election.cc:302] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: aaf6d9dc6f5747bd83d2b9e24a7d679a, b8ffc52e71f045e5ac0f8aa35f3eea38; no voters:
I20240502 01:53:10.150678 20079 raft_consensus.cc:2781] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:10.151136 20079 raft_consensus.cc:686] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 2 LEADER]: Becoming Leader. State: Replica: aaf6d9dc6f5747bd83d2b9e24a7d679a, State: Running, Role: LEADER
I20240502 01:53:10.151635 20079 consensus_queue.cc:237] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } }
I20240502 01:53:10.154474 17081 catalog_manager.cc:5484] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a reported cstate change: term changed from 1 to 2, leader changed from b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193) to aaf6d9dc6f5747bd83d2b9e24a7d679a (127.14.212.198). New cstate: current_term: 2 leader_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } } }
I20240502 01:53:10.197768 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:10.203904 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:10.218350 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:10.221056 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:10.268862 17209 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:10.271944 18598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:10.272004 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:10.360146 17230 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d94d0492ef1d4603b77441d09ad3c6dc"
dest_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59500
I20240502 01:53:10.360355 17230 raft_consensus.cc:595] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:10.423185 20153 raft_consensus.cc:979] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38: : Instructing follower d21ca31c11b746d99a102d116cfad615 to start an election
I20240502 01:53:10.423535 20153 raft_consensus.cc:1067] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Signalling peer d21ca31c11b746d99a102d116cfad615 to start an election
I20240502 01:53:10.424417 18619 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d94d0492ef1d4603b77441d09ad3c6dc"
dest_uuid: "d21ca31c11b746d99a102d116cfad615"
from {username='slave'} at 127.14.212.193:48237
I20240502 01:53:10.424664 18619 raft_consensus.cc:482] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:10.424796 18619 raft_consensus.cc:3037] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:10.427031 18619 raft_consensus.cc:504] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } }
I20240502 01:53:10.427567 18619 leader_election.cc:288] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615 [CANDIDATE]: Term 2 election: Requested vote from peers b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193:44487)
I20240502 01:53:10.428495 17230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94d0492ef1d4603b77441d09ad3c6dc" candidate_uuid: "d21ca31c11b746d99a102d116cfad615" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38"
I20240502 01:53:10.428798 17230 raft_consensus.cc:3032] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:10.428956 17230 raft_consensus.cc:729] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Running, Role: LEADER
I20240502 01:53:10.429286 17230 consensus_queue.cc:260] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } }
I20240502 01:53:10.429725 17230 raft_consensus.cc:3037] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:10.433929 17230 raft_consensus.cc:2454] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d21ca31c11b746d99a102d116cfad615 in term 2.
I20240502 01:53:10.434679 18552 leader_election.cc:302] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b8ffc52e71f045e5ac0f8aa35f3eea38, d21ca31c11b746d99a102d116cfad615; no voters:
I20240502 01:53:10.435149 20077 raft_consensus.cc:2781] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:10.435293 20077 raft_consensus.cc:686] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615 [term 2 LEADER]: Becoming Leader. State: Replica: d21ca31c11b746d99a102d116cfad615, State: Running, Role: LEADER
I20240502 01:53:10.435664 20077 consensus_queue.cc:237] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } }
I20240502 01:53:10.438845 17082 catalog_manager.cc:5484] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615 reported cstate change: term changed from 1 to 2, leader changed from b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193) to d21ca31c11b746d99a102d116cfad615 (127.14.212.196). New cstate: current_term: 2 leader_uuid: "d21ca31c11b746d99a102d116cfad615" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } } }
I20240502 01:53:10.466779 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:10.483002 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:10.486258 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:10.514564 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:10.517136 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:10.530417 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:10.562168 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:10.737723 17230 raft_consensus.cc:1261] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Refusing update from remote peer aaf6d9dc6f5747bd83d2b9e24a7d679a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:53:10.738328 20214 consensus_queue.cc:1035] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:10.905301 17230 raft_consensus.cc:1261] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Refusing update from remote peer d21ca31c11b746d99a102d116cfad615: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:53:10.906440 20077 consensus_queue.cc:1035] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:10.923974 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:10.931813 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:10.942811 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:10.946444 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:10.960726 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:10.962301 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:10.966863 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:11.081696 19016 consensus_queue.cc:237] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } }
I20240502 01:53:11.082983 20005 raft_consensus.cc:2932] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } } }
I20240502 01:53:11.088481 17080 catalog_manager.cc:5484] T 037045b61419472089771a8c6a057d21 P f7baee4920524734aa33ce500a559f3b reported cstate change: config changed from index 4 to 6, VOTER b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193) evicted. New cstate: current_term: 2 leader_uuid: "f7baee4920524734aa33ce500a559f3b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } } }
I20240502 01:53:11.094594 17209 tablet_service.cc:1514] Processing DeleteTablet for tablet 037045b61419472089771a8c6a057d21 with delete_type TABLET_DATA_TOMBSTONED (TS b8ffc52e71f045e5ac0f8aa35f3eea38 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56094
I20240502 01:53:11.095578 20286 tablet_replica.cc:330] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38: stopping tablet replica
I20240502 01:53:11.096024 20286 raft_consensus.cc:2229] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:11.096675 20286 raft_consensus.cc:2258] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:11.098042 20286 ts_tablet_manager.cc:1875] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:11.104517 20286 ts_tablet_manager.cc:1888] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 01:53:11.104686 20286 log.cc:1196] T 037045b61419472089771a8c6a057d21 P b8ffc52e71f045e5ac0f8aa35f3eea38: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/037045b61419472089771a8c6a057d21
I20240502 01:53:11.105824 17073 catalog_manager.cc:4830] TS b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193:44487): tablet 037045b61419472089771a8c6a057d21 (table rebalance_test_table_3 [id=08ac3856c4004b1ba9081159555cd945]) successfully deleted
I20240502 01:53:11.199409 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:11.199409 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:11.207106 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:11.225036 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:11.235433 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:11.247406 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:11.276536 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:11.388722 18884 consensus_queue.cc:237] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } }
I20240502 01:53:11.390313 20319 raft_consensus.cc:2932] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } } }
I20240502 01:53:11.395097 17082 catalog_manager.cc:5484] T b1636e2cd56f40359df64d2ac4c9d806 P aaf6d9dc6f5747bd83d2b9e24a7d679a reported cstate change: config changed from index 4 to 6, VOTER 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194) evicted. New cstate: current_term: 2 leader_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } } }
I20240502 01:53:11.399195 17342 tablet_service.cc:1514] Processing DeleteTablet for tablet b1636e2cd56f40359df64d2ac4c9d806 with delete_type TABLET_DATA_TOMBSTONED (TS 4d190acabedb48b3bbb8cd6b677622da not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46946
I20240502 01:53:11.400139 20340 tablet_replica.cc:330] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da: stopping tablet replica
I20240502 01:53:11.400533 20340 raft_consensus.cc:2229] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:11.400830 20340 raft_consensus.cc:2258] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:11.402050 20340 ts_tablet_manager.cc:1875] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:11.407747 20340 ts_tablet_manager.cc:1888] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 01:53:11.407920 20340 log.cc:1196] T b1636e2cd56f40359df64d2ac4c9d806 P 4d190acabedb48b3bbb8cd6b677622da: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/b1636e2cd56f40359df64d2ac4c9d806
I20240502 01:53:11.409035 17071 catalog_manager.cc:4830] TS 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194:34515): tablet b1636e2cd56f40359df64d2ac4c9d806 (table rebalance_test_table_2 [id=c5068995c1864beca8f4cdbaff1fc3f8]) successfully deleted
I20240502 01:53:11.504010 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:11.522400 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:11.526836 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:11.535115 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:11.553380 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:11.569226 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:11.576355 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:11.670938 18751 consensus_queue.cc:237] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } }
I20240502 01:53:11.672235 20378 raft_consensus.cc:2932] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } } }
I20240502 01:53:11.677580 17083 catalog_manager.cc:5484] T 3fed2519c9b34871b27e5148e18b0673 P adf51a82733d4010b3028cb54cc307e2 reported cstate change: config changed from index 4 to 6, VOTER 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194) evicted. New cstate: current_term: 2 leader_uuid: "adf51a82733d4010b3028cb54cc307e2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } } }
I20240502 01:53:11.681187 17342 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fed2519c9b34871b27e5148e18b0673 with delete_type TABLET_DATA_TOMBSTONED (TS 4d190acabedb48b3bbb8cd6b677622da not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46946
I20240502 01:53:11.681563 20340 tablet_replica.cc:330] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da: stopping tablet replica
I20240502 01:53:11.681752 20340 raft_consensus.cc:2229] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:11.681914 20340 raft_consensus.cc:2258] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:11.682636 20340 ts_tablet_manager.cc:1875] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:11.688709 20340 ts_tablet_manager.cc:1888] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 01:53:11.688809 20340 log.cc:1196] T 3fed2519c9b34871b27e5148e18b0673 P 4d190acabedb48b3bbb8cd6b677622da: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/3fed2519c9b34871b27e5148e18b0673
I20240502 01:53:11.689577 17071 catalog_manager.cc:4830] TS 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194:34515): tablet 3fed2519c9b34871b27e5148e18b0673 (table rebalance_test_table_3 [id=08ac3856c4004b1ba9081159555cd945]) successfully deleted
I20240502 01:53:11.791990 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:11.819584 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:11.826438 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:11.831763 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:11.838563 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:11.845820 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:11.881354 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:11.967298 18751 consensus_queue.cc:237] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } }
I20240502 01:53:11.968521 20396 raft_consensus.cc:2932] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } } }
I20240502 01:53:11.973330 17080 catalog_manager.cc:5484] T 97ffbdb149fb42c892ed094ce60f7166 P adf51a82733d4010b3028cb54cc307e2 reported cstate change: config changed from index 4 to 6, VOTER 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194) evicted. New cstate: current_term: 2 leader_uuid: "adf51a82733d4010b3028cb54cc307e2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "adf51a82733d4010b3028cb54cc307e2" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 41667 } } }
I20240502 01:53:11.977200 17342 tablet_service.cc:1514] Processing DeleteTablet for tablet 97ffbdb149fb42c892ed094ce60f7166 with delete_type TABLET_DATA_TOMBSTONED (TS 4d190acabedb48b3bbb8cd6b677622da not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46946
I20240502 01:53:11.977589 20340 tablet_replica.cc:330] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da: stopping tablet replica
I20240502 01:53:11.977806 20340 raft_consensus.cc:2229] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:11.978049 20340 raft_consensus.cc:2258] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:11.978832 20340 ts_tablet_manager.cc:1875] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:11.984575 20340 ts_tablet_manager.cc:1888] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 01:53:11.984711 20340 log.cc:1196] T 97ffbdb149fb42c892ed094ce60f7166 P 4d190acabedb48b3bbb8cd6b677622da: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/97ffbdb149fb42c892ed094ce60f7166
I20240502 01:53:11.985627 17071 catalog_manager.cc:4830] TS 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194:34515): tablet 97ffbdb149fb42c892ed094ce60f7166 (table rebalance_test_table_4 [id=d01218100b4444e98fec7c8375220687]) successfully deleted
I20240502 01:53:12.084527 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:12.095034 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:12.098884 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:12.109911 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:12.119444 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:12.119925 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:12.129631 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:12.233143 18477 consensus_queue.cc:237] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } }
I20240502 01:53:12.234570 20434 raft_consensus.cc:2932] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } } }
I20240502 01:53:12.239744 17080 catalog_manager.cc:5484] T b477b96739354b6ab43bf4ee230f4559 P d4e4a5c2ba664cd08c3aba1bb348ed75 reported cstate change: config changed from index 4 to 6, VOTER 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194) evicted. New cstate: current_term: 2 leader_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d4e4a5c2ba664cd08c3aba1bb348ed75" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 42281 } } }
I20240502 01:53:12.244143 17342 tablet_service.cc:1514] Processing DeleteTablet for tablet b477b96739354b6ab43bf4ee230f4559 with delete_type TABLET_DATA_TOMBSTONED (TS 4d190acabedb48b3bbb8cd6b677622da not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46946
I20240502 01:53:12.244643 20340 tablet_replica.cc:330] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da: stopping tablet replica
I20240502 01:53:12.244904 20340 raft_consensus.cc:2229] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:12.245169 20340 raft_consensus.cc:2258] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:12.246162 20340 ts_tablet_manager.cc:1875] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:12.252518 20340 ts_tablet_manager.cc:1888] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 01:53:12.252713 20340 log.cc:1196] T b477b96739354b6ab43bf4ee230f4559 P 4d190acabedb48b3bbb8cd6b677622da: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/b477b96739354b6ab43bf4ee230f4559
I20240502 01:53:12.253707 17071 catalog_manager.cc:4830] TS 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194:34515): tablet b477b96739354b6ab43bf4ee230f4559 (table rebalance_test_table_1 [id=3f2e7f170c6a46d283b6e6cd5b54201e]) successfully deleted
I20240502 01:53:12.367480 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:12.370934 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:12.375808 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:12.385290 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:12.400336 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:12.403357 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:12.431988 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:12.531481 19016 consensus_queue.cc:237] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } }
I20240502 01:53:12.533172 20005 raft_consensus.cc:2932] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } } }
I20240502 01:53:12.538717 17081 catalog_manager.cc:5484] T bc1d0f4745df4203848a1f4820f9b499 P f7baee4920524734aa33ce500a559f3b reported cstate change: config changed from index 4 to 6, VOTER b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193) evicted. New cstate: current_term: 2 leader_uuid: "f7baee4920524734aa33ce500a559f3b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f7baee4920524734aa33ce500a559f3b" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 45747 } } }
I20240502 01:53:12.542759 17209 tablet_service.cc:1514] Processing DeleteTablet for tablet bc1d0f4745df4203848a1f4820f9b499 with delete_type TABLET_DATA_TOMBSTONED (TS b8ffc52e71f045e5ac0f8aa35f3eea38 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56094
I20240502 01:53:12.543524 20564 tablet_replica.cc:330] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38: stopping tablet replica
I20240502 01:53:12.543759 20564 raft_consensus.cc:2229] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:12.543982 20564 raft_consensus.cc:2258] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:12.544651 20564 ts_tablet_manager.cc:1875] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:12.549767 20564 ts_tablet_manager.cc:1888] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 01:53:12.549908 20564 log.cc:1196] T bc1d0f4745df4203848a1f4820f9b499 P b8ffc52e71f045e5ac0f8aa35f3eea38: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/bc1d0f4745df4203848a1f4820f9b499
I20240502 01:53:12.550767 17073 catalog_manager.cc:4830] TS b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193:44487): tablet bc1d0f4745df4203848a1f4820f9b499 (table rebalance_test_table_4 [id=d01218100b4444e98fec7c8375220687]) successfully deleted
I20240502 01:53:12.671398 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:12.677486 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:12.686594 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:12.688705 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:12.698423 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:12.700855 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:12.754520 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:12.844955 18619 consensus_queue.cc:237] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } }
I20240502 01:53:12.846392 20077 raft_consensus.cc:2932] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } } }
I20240502 01:53:12.851644 17080 catalog_manager.cc:5484] T e4a22d109d8044adaeb364037e9c137e P d21ca31c11b746d99a102d116cfad615 reported cstate change: config changed from index 4 to 6, VOTER b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193) evicted. New cstate: current_term: 2 leader_uuid: "d21ca31c11b746d99a102d116cfad615" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } } }
I20240502 01:53:12.855564 17209 tablet_service.cc:1514] Processing DeleteTablet for tablet e4a22d109d8044adaeb364037e9c137e with delete_type TABLET_DATA_TOMBSTONED (TS b8ffc52e71f045e5ac0f8aa35f3eea38 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56094
I20240502 01:53:12.855990 20564 tablet_replica.cc:330] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38: stopping tablet replica
I20240502 01:53:12.856274 20564 raft_consensus.cc:2229] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:12.856503 20564 raft_consensus.cc:2258] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:12.857228 20564 ts_tablet_manager.cc:1875] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:12.862241 20564 ts_tablet_manager.cc:1888] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 01:53:12.862434 20564 log.cc:1196] T e4a22d109d8044adaeb364037e9c137e P b8ffc52e71f045e5ac0f8aa35f3eea38: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/e4a22d109d8044adaeb364037e9c137e
I20240502 01:53:12.863299 17073 catalog_manager.cc:4830] TS b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193:44487): tablet e4a22d109d8044adaeb364037e9c137e (table rebalance_test_table_2 [id=c5068995c1864beca8f4cdbaff1fc3f8]) successfully deleted
I20240502 01:53:12.968843 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:12.969002 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:12.991737 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:12.995992 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:13.030936 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:13.032030 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:13.047343 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:13.124135 18884 consensus_queue.cc:237] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } }
I20240502 01:53:13.125453 20624 raft_consensus.cc:2932] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } } }
I20240502 01:53:13.130954 17081 catalog_manager.cc:5484] T 1e8092e428084a56bbfaaca963877631 P aaf6d9dc6f5747bd83d2b9e24a7d679a reported cstate change: config changed from index 4 to 6, VOTER b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193) evicted. New cstate: current_term: 2 leader_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aaf6d9dc6f5747bd83d2b9e24a7d679a" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 34487 } } }
I20240502 01:53:13.135129 17209 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e8092e428084a56bbfaaca963877631 with delete_type TABLET_DATA_TOMBSTONED (TS b8ffc52e71f045e5ac0f8aa35f3eea38 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56094
I20240502 01:53:13.135495 20564 tablet_replica.cc:330] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38: stopping tablet replica
I20240502 01:53:13.135691 20564 raft_consensus.cc:2229] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:13.135902 20564 raft_consensus.cc:2258] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:13.136636 20564 ts_tablet_manager.cc:1875] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:13.142037 20564 ts_tablet_manager.cc:1888] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 01:53:13.142189 20564 log.cc:1196] T 1e8092e428084a56bbfaaca963877631 P b8ffc52e71f045e5ac0f8aa35f3eea38: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/1e8092e428084a56bbfaaca963877631
I20240502 01:53:13.143055 17073 catalog_manager.cc:4830] TS b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193:44487): tablet 1e8092e428084a56bbfaaca963877631 (table rebalance_test_table_0 [id=edade0db079c4e7b9fa4c7a565260d19]) successfully deleted
I20240502 01:53:13.246726 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:13.259817 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:13.262737 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:13.284080 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:13.295704 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:13.320104 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:13.322533 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:13.421042 18619 consensus_queue.cc:237] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } }
I20240502 01:53:13.422447 20077 raft_consensus.cc:2932] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } } }
I20240502 01:53:13.427508 17080 catalog_manager.cc:5484] T d94d0492ef1d4603b77441d09ad3c6dc P d21ca31c11b746d99a102d116cfad615 reported cstate change: config changed from index 4 to 6, VOTER b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193) evicted. New cstate: current_term: 2 leader_uuid: "d21ca31c11b746d99a102d116cfad615" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d21ca31c11b746d99a102d116cfad615" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43697 } } }
I20240502 01:53:13.431459 17209 tablet_service.cc:1514] Processing DeleteTablet for tablet d94d0492ef1d4603b77441d09ad3c6dc with delete_type TABLET_DATA_TOMBSTONED (TS b8ffc52e71f045e5ac0f8aa35f3eea38 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56094
I20240502 01:53:13.431872 20564 tablet_replica.cc:330] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38: stopping tablet replica
I20240502 01:53:13.432065 20564 raft_consensus.cc:2229] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:13.432264 20564 raft_consensus.cc:2258] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:13.432914 20564 ts_tablet_manager.cc:1875] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:13.437767 20564 ts_tablet_manager.cc:1888] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 01:53:13.437894 20564 log.cc:1196] T d94d0492ef1d4603b77441d09ad3c6dc P b8ffc52e71f045e5ac0f8aa35f3eea38: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/d94d0492ef1d4603b77441d09ad3c6dc
I20240502 01:53:13.438659 17073 catalog_manager.cc:4830] TS b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193:44487): tablet d94d0492ef1d4603b77441d09ad3c6dc (table rebalance_test_table_0 [id=edade0db079c4e7b9fa4c7a565260d19]) successfully deleted
I20240502 01:53:13.853726 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:13.866896 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:13.877166 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:13.878015 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:13.879523 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:13.889708 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:13.910327 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:14.088376 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:14.092411 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:14.101133 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:14.101444 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:14.102545 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:14.118997 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:14.124558 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:14.915489 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:14.922458 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:14.927678 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:14.932777 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:14.950096 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:14.960635 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:14.977214 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:15.192029 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:15.204782 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:15.217366 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:15.217602 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:15.235643 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:15.259857 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:15.292958 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:15.575783 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:15.591306 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:15.602703 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:15.621229 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:15.623116 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:15.624034 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:15.664320 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
0939f6abb7634ecd9c58879083ac7987 | 127.14.212.254:43511 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.14.212.212:37613 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
4d190acabedb48b3bbb8cd6b677622da | 127.14.212.194:34515 | HEALTHY | <none> | 3 | 0
aaf6d9dc6f5747bd83d2b9e24a7d679a | 127.14.212.198:34487 | HEALTHY | <none> | 2 | 0
adf51a82733d4010b3028cb54cc307e2 | 127.14.212.197:41667 | HEALTHY | <none> | 2 | 0
b8ffc52e71f045e5ac0f8aa35f3eea38 | 127.14.212.193:44487 | HEALTHY | <none> | 3 | 0
d21ca31c11b746d99a102d116cfad615 | 127.14.212.196:43697 | HEALTHY | <none> | 2 | 0
d4e4a5c2ba664cd08c3aba1bb348ed75 | 127.14.212.195:42281 | HEALTHY | <none> | 2 | 0
f7baee4920524734aa33ce500a559f3b | 127.14.212.199:45747 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.14.212.193 | experimental | 127.14.212.193:44487
local_ip_for_outbound_sockets | 127.14.212.194 | experimental | 127.14.212.194:34515
local_ip_for_outbound_sockets | 127.14.212.195 | experimental | 127.14.212.195:42281
local_ip_for_outbound_sockets | 127.14.212.196 | experimental | 127.14.212.196:43697
local_ip_for_outbound_sockets | 127.14.212.197 | experimental | 127.14.212.197:41667
local_ip_for_outbound_sockets | 127.14.212.198 | experimental | 127.14.212.198:34487
local_ip_for_outbound_sockets | 127.14.212.199 | experimental | 127.14.212.199:45747
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.14.212.193:44487
server_dump_info_path | /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.14.212.194:34515
server_dump_info_path | /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.14.212.195:42281
server_dump_info_path | /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.14.212.196:43697
server_dump_info_path | /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.14.212.197:41667
server_dump_info_path | /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.14.212.198:34487
server_dump_info_path | /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.14.212.199:45747
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.14.212.212:37613 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 01:53:15.717155 15187 log_verifier.cc:126] Checking tablet 037045b61419472089771a8c6a057d21
I20240502 01:53:15.734413 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 037045b61419472089771a8c6a057d21
I20240502 01:53:15.734597 15187 log_verifier.cc:126] Checking tablet 149a3a4584fe4c82a59ca4bc28ca80dd
I20240502 01:53:15.748307 15187 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 149a3a4584fe4c82a59ca4bc28ca80dd
I20240502 01:53:15.748425 15187 log_verifier.cc:126] Checking tablet 1e8092e428084a56bbfaaca963877631
I20240502 01:53:15.761435 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 1e8092e428084a56bbfaaca963877631
I20240502 01:53:15.761569 15187 log_verifier.cc:126] Checking tablet 2bfc0209bd434fa9a56b45d7a9b73a12
I20240502 01:53:15.774322 15187 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2bfc0209bd434fa9a56b45d7a9b73a12
I20240502 01:53:15.774434 15187 log_verifier.cc:126] Checking tablet 3fed2519c9b34871b27e5148e18b0673
I20240502 01:53:15.787268 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 3fed2519c9b34871b27e5148e18b0673
I20240502 01:53:15.787389 15187 log_verifier.cc:126] Checking tablet 4d4eaec689d345b4a812a448e5d359e4
I20240502 01:53:15.800027 15187 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4d4eaec689d345b4a812a448e5d359e4
I20240502 01:53:15.800161 15187 log_verifier.cc:126] Checking tablet 97ffbdb149fb42c892ed094ce60f7166
I20240502 01:53:15.812829 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 97ffbdb149fb42c892ed094ce60f7166
I20240502 01:53:15.812994 15187 log_verifier.cc:126] Checking tablet a487f57191fe4eb998ad8f5f86f4050b
I20240502 01:53:15.820770 15187 log_verifier.cc:177] Verified matching terms for 2 ops in tablet a487f57191fe4eb998ad8f5f86f4050b
I20240502 01:53:15.820919 15187 log_verifier.cc:126] Checking tablet b1636e2cd56f40359df64d2ac4c9d806
I20240502 01:53:15.833721 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b1636e2cd56f40359df64d2ac4c9d806
I20240502 01:53:15.833855 15187 log_verifier.cc:126] Checking tablet b477b96739354b6ab43bf4ee230f4559
I20240502 01:53:15.846586 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b477b96739354b6ab43bf4ee230f4559
I20240502 01:53:15.846721 15187 log_verifier.cc:126] Checking tablet ba6e000d044241edbb347c80dca80b58
I20240502 01:53:15.859899 15187 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ba6e000d044241edbb347c80dca80b58
I20240502 01:53:15.860026 15187 log_verifier.cc:126] Checking tablet bc1d0f4745df4203848a1f4820f9b499
I20240502 01:53:15.875478 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet bc1d0f4745df4203848a1f4820f9b499
I20240502 01:53:15.875645 15187 log_verifier.cc:126] Checking tablet bf441bb1e6064b56b745ffe8d02a3983
I20240502 01:53:15.890712 15187 log_verifier.cc:177] Verified matching terms for 3 ops in tablet bf441bb1e6064b56b745ffe8d02a3983
I20240502 01:53:15.890842 15187 log_verifier.cc:126] Checking tablet c130cb706b004c2088925fba192a8c90
I20240502 01:53:15.905476 15187 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c130cb706b004c2088925fba192a8c90
I20240502 01:53:15.905620 15187 log_verifier.cc:126] Checking tablet d94d0492ef1d4603b77441d09ad3c6dc
I20240502 01:53:15.918273 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d94d0492ef1d4603b77441d09ad3c6dc
I20240502 01:53:15.918416 15187 log_verifier.cc:126] Checking tablet e4a22d109d8044adaeb364037e9c137e
I20240502 01:53:15.932344 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e4a22d109d8044adaeb364037e9c137e
I20240502 01:53:15.935371 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.200:0
--local_ip_for_outbound_sockets=127.14.212.200
--webserver_interface=127.14.212.200
--webserver_port=0
--tserver_master_addrs=127.14.212.254:43511
--builtin_ntp_servers=127.14.212.212:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:53:16.307451 20950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:16.307763 20950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:16.307891 20950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:16.315934 20950 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:16.316076 20950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:16.316243 20950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.200
I20240502 01:53:16.332346 20950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.14.212.200
--webserver_port=0
--tserver_master_addrs=127.14.212.254:43511
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.200
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:16.334115 20950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:16.336225 20950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:16.350728 20955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:16.351248 20956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:16.351835 20958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:16.352723 20950 server_base.cc:1034] running on GCE node
I20240502 01:53:16.353416 20950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:16.354480 20950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:16.355746 20950 hybrid_clock.cc:648] HybridClock initialized: now 1714614796355691 us; error 45 us; skew 500 ppm
I20240502 01:53:16.356160 20950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:16.359088 20950 webserver.cc:460] Webserver started at http://127.14.212.200:42987/ using document root <none> and password file <none>
I20240502 01:53:16.359918 20950 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:16.360023 20950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:16.360311 20950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:16.363224 20950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a"
format_stamp: "Formatted at 2024-05-02 01:53:16 on dist-test-slave-mk1r"
I20240502 01:53:16.364081 20950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a"
format_stamp: "Formatted at 2024-05-02 01:53:16 on dist-test-slave-mk1r"
I20240502 01:53:16.369902 20950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240502 01:53:16.373732 20964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:16.375263 20950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:53:16.375473 20950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a"
format_stamp: "Formatted at 2024-05-02 01:53:16 on dist-test-slave-mk1r"
I20240502 01:53:16.375717 20950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:16.398066 20950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:16.399147 20950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:16.399470 20950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:16.400599 20950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:16.402374 20950 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:16.402472 20950 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:16.402597 20950 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:16.402815 20950 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:16.444396 20950 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.200:41413
I20240502 01:53:16.444465 21078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.200:41413 every 8 connection(s)
I20240502 01:53:16.445950 20950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:53:16.448331 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 20950
I20240502 01:53:16.448560 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 01:53:16.463280 21079 heartbeater.cc:347] Connected to a master server at 127.14.212.254:43511
I20240502 01:53:16.463677 21079 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:16.464903 21079 heartbeater.cc:511] Master 127.14.212.254:43511 requested a full tablet report, sending...
I20240502 01:53:16.466758 17084 ts_manager.cc:194] Registered new tserver with Master: 3b4ab852d7db4fb09b81f798b0dbaa2a (127.14.212.200:41413)
I20240502 01:53:16.467583 17084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.200:46479
I20240502 01:53:17.037266 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:17.053165 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:17.053454 21014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:17.056237 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:17.066711 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:17.072089 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:17.075898 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:17.076383 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:17.392071 21014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:17.402784 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:17.412211 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:17.416698 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:17.424196 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:17.440963 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:17.449549 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:17.458495 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:17.471119 21079 heartbeater.cc:503] Master 127.14.212.254:43511 was elected leader, sending a full tablet report...
I20240502 01:53:17.565667 17230 consensus_queue.cc:237] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } }
W20240502 01:53:17.574935 17162 consensus_peers.cc:485] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 -> Peer 3b4ab852d7db4fb09b81f798b0dbaa2a (127.14.212.200:41413): Couldn't send request to peer 3b4ab852d7db4fb09b81f798b0dbaa2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf441bb1e6064b56b745ffe8d02a3983. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:18.008409 21187 ts_tablet_manager.cc:902] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a: Initiating tablet copy from peer b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193:44487)
I20240502 01:53:18.010998 21187 tablet_copy_client.cc:308] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.193:44487
I20240502 01:53:18.017748 17250 tablet_copy_service.cc:140] P b8ffc52e71f045e5ac0f8aa35f3eea38: Received BeginTabletCopySession request for tablet bf441bb1e6064b56b745ffe8d02a3983 from peer 3b4ab852d7db4fb09b81f798b0dbaa2a ({username='slave'} at 127.14.212.200:48939)
I20240502 01:53:18.017941 17250 tablet_copy_service.cc:161] P b8ffc52e71f045e5ac0f8aa35f3eea38: Beginning new tablet copy session on tablet bf441bb1e6064b56b745ffe8d02a3983 from peer 3b4ab852d7db4fb09b81f798b0dbaa2a at {username='slave'} at 127.14.212.200:48939: session id = 3b4ab852d7db4fb09b81f798b0dbaa2a-bf441bb1e6064b56b745ffe8d02a3983
I20240502 01:53:18.019925 17250 tablet_copy_source_session.cc:205] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:18.024458 21187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf441bb1e6064b56b745ffe8d02a3983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:18.034054 21187 tablet_copy_client.cc:783] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:18.034353 21187 tablet_copy_client.cc:647] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:18.036613 21187 tablet_copy_client.cc:523] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:18.039788 21187 tablet_bootstrap.cc:492] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a: Bootstrap starting.
I20240502 01:53:18.049490 21187 log.cc:824] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:18.082468 21187 tablet_bootstrap.cc:492] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:18.083024 21187 tablet_bootstrap.cc:492] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a: Bootstrap complete.
I20240502 01:53:18.083470 21187 ts_tablet_manager.cc:1367] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a: Time spent bootstrapping tablet: real 0.044s user 0.032s sys 0.011s
I20240502 01:53:18.088165 21187 raft_consensus.cc:348] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } }
I20240502 01:53:18.089068 21187 raft_consensus.cc:729] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4ab852d7db4fb09b81f798b0dbaa2a, State: Initialized, Role: FOLLOWER
I20240502 01:53:18.089504 21187 consensus_queue.cc:260] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } }
I20240502 01:53:18.090939 21187 ts_tablet_manager.cc:1398] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20240502 01:53:18.092096 17250 tablet_copy_service.cc:342] P b8ffc52e71f045e5ac0f8aa35f3eea38: Request end of tablet copy session 3b4ab852d7db4fb09b81f798b0dbaa2a-bf441bb1e6064b56b745ffe8d02a3983 received from {username='slave'} at 127.14.212.200:48939
I20240502 01:53:18.092226 17250 tablet_copy_service.cc:433] P b8ffc52e71f045e5ac0f8aa35f3eea38: ending tablet copy session 3b4ab852d7db4fb09b81f798b0dbaa2a-bf441bb1e6064b56b745ffe8d02a3983 on tablet bf441bb1e6064b56b745ffe8d02a3983 with peer 3b4ab852d7db4fb09b81f798b0dbaa2a
I20240502 01:53:18.398468 21034 raft_consensus.cc:1203] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:18.400379 21195 raft_consensus.cc:2932] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b4ab852d7db4fb09b81f798b0dbaa2a (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } } }
I20240502 01:53:18.401172 21034 raft_consensus.cc:2932] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b4ab852d7db4fb09b81f798b0dbaa2a (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } } }
I20240502 01:53:18.406303 17084 catalog_manager.cc:5484] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 reported cstate change: config changed from index -1 to 4, VOTER 3b4ab852d7db4fb09b81f798b0dbaa2a (127.14.212.200) added. New cstate: current_term: 1 leader_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } } }
I20240502 01:53:18.439533 17361 consensus_queue.cc:237] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } }
W20240502 01:53:18.453462 17295 consensus_peers.cc:485] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da -> Peer 3b4ab852d7db4fb09b81f798b0dbaa2a (127.14.212.200:41413): Couldn't send request to peer 3b4ab852d7db4fb09b81f798b0dbaa2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba6e000d044241edbb347c80dca80b58. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:18.977438 21224 ts_tablet_manager.cc:902] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a: Initiating tablet copy from peer 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194:34515)
I20240502 01:53:18.978530 21224 tablet_copy_client.cc:308] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:34515
I20240502 01:53:18.984992 17383 tablet_copy_service.cc:140] P 4d190acabedb48b3bbb8cd6b677622da: Received BeginTabletCopySession request for tablet ba6e000d044241edbb347c80dca80b58 from peer 3b4ab852d7db4fb09b81f798b0dbaa2a ({username='slave'} at 127.14.212.200:51599)
I20240502 01:53:18.985148 17383 tablet_copy_service.cc:161] P 4d190acabedb48b3bbb8cd6b677622da: Beginning new tablet copy session on tablet ba6e000d044241edbb347c80dca80b58 from peer 3b4ab852d7db4fb09b81f798b0dbaa2a at {username='slave'} at 127.14.212.200:51599: session id = 3b4ab852d7db4fb09b81f798b0dbaa2a-ba6e000d044241edbb347c80dca80b58
I20240502 01:53:18.987422 17383 tablet_copy_source_session.cc:205] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:18.988749 21224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6e000d044241edbb347c80dca80b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:18.993458 21224 tablet_copy_client.cc:783] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:18.993703 21224 tablet_copy_client.cc:647] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:18.995316 21224 tablet_copy_client.cc:523] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:18.997860 21224 tablet_bootstrap.cc:492] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a: Bootstrap starting.
I20240502 01:53:19.028529 21224 tablet_bootstrap.cc:492] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:19.029004 21224 tablet_bootstrap.cc:492] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a: Bootstrap complete.
I20240502 01:53:19.029243 21224 ts_tablet_manager.cc:1367] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.012s
I20240502 01:53:19.030077 21224 raft_consensus.cc:348] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } }
I20240502 01:53:19.030428 21224 raft_consensus.cc:729] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4ab852d7db4fb09b81f798b0dbaa2a, State: Initialized, Role: FOLLOWER
I20240502 01:53:19.030575 21224 consensus_queue.cc:260] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } }
I20240502 01:53:19.031275 21224 ts_tablet_manager.cc:1398] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:19.032147 17383 tablet_copy_service.cc:342] P 4d190acabedb48b3bbb8cd6b677622da: Request end of tablet copy session 3b4ab852d7db4fb09b81f798b0dbaa2a-ba6e000d044241edbb347c80dca80b58 received from {username='slave'} at 127.14.212.200:51599
I20240502 01:53:19.032325 17383 tablet_copy_service.cc:433] P 4d190acabedb48b3bbb8cd6b677622da: ending tablet copy session 3b4ab852d7db4fb09b81f798b0dbaa2a-ba6e000d044241edbb347c80dca80b58 on tablet ba6e000d044241edbb347c80dca80b58 with peer 3b4ab852d7db4fb09b81f798b0dbaa2a
I20240502 01:53:19.441341 21034 raft_consensus.cc:1203] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:19.442793 21233 raft_consensus.cc:2932] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b4ab852d7db4fb09b81f798b0dbaa2a (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } } }
I20240502 01:53:19.443437 21034 raft_consensus.cc:2932] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3b4ab852d7db4fb09b81f798b0dbaa2a (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } } }
I20240502 01:53:19.448160 17084 catalog_manager.cc:5484] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a reported cstate change: config changed from index -1 to 4, VOTER 3b4ab852d7db4fb09b81f798b0dbaa2a (127.14.212.200) added. New cstate: current_term: 1 leader_uuid: "4d190acabedb48b3bbb8cd6b677622da" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } } }
I20240502 01:53:19.448699 21233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:45996 (request call id 0) took 1009 ms. Trace:
I20240502 01:53:19.448872 21233 rpcz_store.cc:276] 0502 01:53:18.438982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 01:53:18.439048 (+ 66us) service_pool.cc:225] Handling call
0502 01:53:18.440281 (+ 1233us) log.cc:842] Serialized 263 byte log entry
0502 01:53:19.448647 (+1008366us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":79,"raft.queue_time_us":572,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":283,"thread_start_us":719,"threads_started":2,"wal-append.queue_time_us":379,"wal-append.run_cpu_time_us":712,"wal-append.run_wall_time_us":1000874}
I20240502 01:53:19.575507 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:19.577726 21014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:19.595446 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:19.606139 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:19.611570 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:19.630892 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:19.644495 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:19.662029 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:19.732875 17361 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ba6e000d044241edbb347c80dca80b58"
dest_uuid: "4d190acabedb48b3bbb8cd6b677622da"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36310
I20240502 01:53:19.733161 17361 raft_consensus.cc:595] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:19.838923 21014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:19.845058 17342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:19.856379 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:19.866650 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:19.869828 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:19.872530 21223 raft_consensus.cc:979] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da: : Instructing follower 3b4ab852d7db4fb09b81f798b0dbaa2a to start an election
I20240502 01:53:19.872718 21233 raft_consensus.cc:1067] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Signalling peer 3b4ab852d7db4fb09b81f798b0dbaa2a to start an election
I20240502 01:53:19.874202 21034 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ba6e000d044241edbb347c80dca80b58"
dest_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a"
from {username='slave'} at 127.14.212.194:47669
I20240502 01:53:19.874563 21034 raft_consensus.cc:482] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:19.874768 21034 raft_consensus.cc:3037] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:19.878063 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:19.887746 21034 raft_consensus.cc:504] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } }
I20240502 01:53:19.891346 21034 leader_election.cc:288] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a [CANDIDATE]: Term 2 election: Requested vote from peers 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194:34515)
I20240502 01:53:19.893350 17361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6e000d044241edbb347c80dca80b58" candidate_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4d190acabedb48b3bbb8cd6b677622da"
I20240502 01:53:19.893818 17361 raft_consensus.cc:3032] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:19.893954 17361 raft_consensus.cc:729] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d190acabedb48b3bbb8cd6b677622da, State: Running, Role: LEADER
I20240502 01:53:19.894335 17361 consensus_queue.cc:260] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } }
I20240502 01:53:19.894657 17361 raft_consensus.cc:3037] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:19.897768 17361 raft_consensus.cc:2454] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b4ab852d7db4fb09b81f798b0dbaa2a in term 2.
I20240502 01:53:19.898646 20965 leader_election.cc:302] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b4ab852d7db4fb09b81f798b0dbaa2a, 4d190acabedb48b3bbb8cd6b677622da; no voters:
I20240502 01:53:19.899366 21232 raft_consensus.cc:2781] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:19.908416 21232 raft_consensus.cc:686] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 2 LEADER]: Becoming Leader. State: Replica: 3b4ab852d7db4fb09b81f798b0dbaa2a, State: Running, Role: LEADER
I20240502 01:53:19.909399 21232 consensus_queue.cc:237] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } }
I20240502 01:53:19.918185 17078 catalog_manager.cc:5484] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a reported cstate change: term changed from 1 to 2, leader changed from 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194) to 3b4ab852d7db4fb09b81f798b0dbaa2a (127.14.212.200). New cstate: current_term: 2 leader_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 } } peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } } }
I20240502 01:53:19.924957 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:19.942077 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.127388 17342 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.135613 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.140733 21014 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:20.143000 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.150552 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.153710 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.156760 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.165390 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:20.284021 17230 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bf441bb1e6064b56b745ffe8d02a3983"
dest_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35910
I20240502 01:53:20.284293 17230 raft_consensus.cc:595] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:20.381722 21014 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:20.387724 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.392434 17342 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.405982 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.417860 17209 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:20.437062 17361 raft_consensus.cc:1261] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Refusing update from remote peer 3b4ab852d7db4fb09b81f798b0dbaa2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:53:20.438520 21410 consensus_queue.cc:1035] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d190acabedb48b3bbb8cd6b677622da" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 34515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:20.466950 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.470371 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.492149 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.545367 21418 raft_consensus.cc:979] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38: : Instructing follower 3b4ab852d7db4fb09b81f798b0dbaa2a to start an election
I20240502 01:53:20.545701 21417 raft_consensus.cc:1067] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Signalling peer 3b4ab852d7db4fb09b81f798b0dbaa2a to start an election
I20240502 01:53:20.546718 21034 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bf441bb1e6064b56b745ffe8d02a3983"
dest_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a"
from {username='slave'} at 127.14.212.193:48037
I20240502 01:53:20.547004 21034 raft_consensus.cc:482] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:20.547132 21034 raft_consensus.cc:3037] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:20.550324 21034 raft_consensus.cc:504] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } }
I20240502 01:53:20.551781 17230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf441bb1e6064b56b745ffe8d02a3983" candidate_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38"
I20240502 01:53:20.552047 17230 raft_consensus.cc:3032] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:20.552150 17230 raft_consensus.cc:729] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b8ffc52e71f045e5ac0f8aa35f3eea38, State: Running, Role: LEADER
I20240502 01:53:20.552475 17230 consensus_queue.cc:260] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } }
I20240502 01:53:20.552707 17230 raft_consensus.cc:3037] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:20.555840 17230 raft_consensus.cc:2454] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b4ab852d7db4fb09b81f798b0dbaa2a in term 2.
I20240502 01:53:20.556213 21034 leader_election.cc:288] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a [CANDIDATE]: Term 2 election: Requested vote from peers b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193:44487)
I20240502 01:53:20.556524 20967 leader_election.cc:302] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b4ab852d7db4fb09b81f798b0dbaa2a, b8ffc52e71f045e5ac0f8aa35f3eea38; no voters:
I20240502 01:53:20.556902 21411 raft_consensus.cc:2781] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:20.557063 21411 raft_consensus.cc:686] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 2 LEADER]: Becoming Leader. State: Replica: 3b4ab852d7db4fb09b81f798b0dbaa2a, State: Running, Role: LEADER
I20240502 01:53:20.557394 21411 consensus_queue.cc:237] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } }
I20240502 01:53:20.560209 17084 catalog_manager.cc:5484] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a reported cstate change: term changed from 1 to 2, leader changed from b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193) to 3b4ab852d7db4fb09b81f798b0dbaa2a (127.14.212.200). New cstate: current_term: 2 leader_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 } } peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } } }
I20240502 01:53:20.897466 21014 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.899189 17342 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.918323 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.924449 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.928737 17209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.940532 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.957259 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:20.968525 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.055267 21034 consensus_queue.cc:237] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } }
I20240502 01:53:21.056169 21411 raft_consensus.cc:2932] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } } }
I20240502 01:53:21.061067 17080 catalog_manager.cc:5484] T ba6e000d044241edbb347c80dca80b58 P 3b4ab852d7db4fb09b81f798b0dbaa2a reported cstate change: config changed from index 4 to 6, VOTER 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194) evicted. New cstate: current_term: 2 leader_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } } }
I20240502 01:53:21.065085 17342 tablet_service.cc:1514] Processing DeleteTablet for tablet ba6e000d044241edbb347c80dca80b58 with delete_type TABLET_DATA_TOMBSTONED (TS 4d190acabedb48b3bbb8cd6b677622da not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46946
I20240502 01:53:21.065704 21494 tablet_replica.cc:330] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da: stopping tablet replica
I20240502 01:53:21.065902 21494 raft_consensus.cc:2229] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:21.066097 21494 raft_consensus.cc:2258] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:21.067896 21494 ts_tablet_manager.cc:1875] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:21.073462 21494 ts_tablet_manager.cc:1888] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 01:53:21.073570 21494 log.cc:1196] T ba6e000d044241edbb347c80dca80b58 P 4d190acabedb48b3bbb8cd6b677622da: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/ba6e000d044241edbb347c80dca80b58
I20240502 01:53:21.074443 17071 catalog_manager.cc:4830] TS 4d190acabedb48b3bbb8cd6b677622da (127.14.212.194:34515): tablet ba6e000d044241edbb347c80dca80b58 (table rebalance_test_table_3 [id=08ac3856c4004b1ba9081159555cd945]) successfully deleted
I20240502 01:53:21.078634 17230 raft_consensus.cc:1261] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Refusing update from remote peer 3b4ab852d7db4fb09b81f798b0dbaa2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:53:21.079293 21488 consensus_queue.cc:1035] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8ffc52e71f045e5ac0f8aa35f3eea38" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 44487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:21.168992 21014 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.183503 17209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.184625 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.184748 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.186726 17342 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.193575 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.209107 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.232038 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.350904 21034 consensus_queue.cc:237] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } }
I20240502 01:53:21.351783 21411 raft_consensus.cc:2932] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } } }
I20240502 01:53:21.356068 17081 catalog_manager.cc:5484] T bf441bb1e6064b56b745ffe8d02a3983 P 3b4ab852d7db4fb09b81f798b0dbaa2a reported cstate change: config changed from index 4 to 6, VOTER b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193) evicted. New cstate: current_term: 2 leader_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3b4ab852d7db4fb09b81f798b0dbaa2a" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 41413 } } }
I20240502 01:53:21.359922 17209 tablet_service.cc:1514] Processing DeleteTablet for tablet bf441bb1e6064b56b745ffe8d02a3983 with delete_type TABLET_DATA_TOMBSTONED (TS b8ffc52e71f045e5ac0f8aa35f3eea38 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56094
I20240502 01:53:21.360739 21547 tablet_replica.cc:330] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38: stopping tablet replica
I20240502 01:53:21.360973 21547 raft_consensus.cc:2229] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:21.361177 21547 raft_consensus.cc:2258] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:21.362735 21547 ts_tablet_manager.cc:1875] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:21.367173 21547 ts_tablet_manager.cc:1888] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 01:53:21.367291 21547 log.cc:1196] T bf441bb1e6064b56b745ffe8d02a3983 P b8ffc52e71f045e5ac0f8aa35f3eea38: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/bf441bb1e6064b56b745ffe8d02a3983
I20240502 01:53:21.368088 17073 catalog_manager.cc:4830] TS b8ffc52e71f045e5ac0f8aa35f3eea38 (127.14.212.193:44487): tablet bf441bb1e6064b56b745ffe8d02a3983 (table rebalance_test_table_2 [id=c5068995c1864beca8f4cdbaff1fc3f8]) successfully deleted
I20240502 01:53:21.688165 21014 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.700196 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.701431 17342 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.715806 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.715939 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.743726 17209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.766614 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.784646 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.946213 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.946554 17342 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.947268 21014 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.961971 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.964112 17209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.984160 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.989842 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:21.992326 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:22.313443 21014 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:22.318825 18730 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:22.336555 18864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:22.346305 17342 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:22.376212 17209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:22.376776 18598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:22.407222 18455 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:22.442682 18996 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
0939f6abb7634ecd9c58879083ac7987 | 127.14.212.254:43511 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.14.212.212:37613 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
3b4ab852d7db4fb09b81f798b0dbaa2a | 127.14.212.200:41413 | HEALTHY | <none> | 2 | 0
4d190acabedb48b3bbb8cd6b677622da | 127.14.212.194:34515 | HEALTHY | <none> | 2 | 0
aaf6d9dc6f5747bd83d2b9e24a7d679a | 127.14.212.198:34487 | HEALTHY | <none> | 2 | 0
adf51a82733d4010b3028cb54cc307e2 | 127.14.212.197:41667 | HEALTHY | <none> | 2 | 0
b8ffc52e71f045e5ac0f8aa35f3eea38 | 127.14.212.193:44487 | HEALTHY | <none> | 2 | 0
d21ca31c11b746d99a102d116cfad615 | 127.14.212.196:43697 | HEALTHY | <none> | 2 | 0
d4e4a5c2ba664cd08c3aba1bb348ed75 | 127.14.212.195:42281 | HEALTHY | <none> | 2 | 0
f7baee4920524734aa33ce500a559f3b | 127.14.212.199:45747 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.14.212.193 | experimental | 127.14.212.193:44487
local_ip_for_outbound_sockets | 127.14.212.194 | experimental | 127.14.212.194:34515
local_ip_for_outbound_sockets | 127.14.212.195 | experimental | 127.14.212.195:42281
local_ip_for_outbound_sockets | 127.14.212.196 | experimental | 127.14.212.196:43697
local_ip_for_outbound_sockets | 127.14.212.197 | experimental | 127.14.212.197:41667
local_ip_for_outbound_sockets | 127.14.212.198 | experimental | 127.14.212.198:34487
local_ip_for_outbound_sockets | 127.14.212.199 | experimental | 127.14.212.199:45747
local_ip_for_outbound_sockets | 127.14.212.200 | experimental | 127.14.212.200:41413
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.14.212.193:44487
server_dump_info_path | /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.14.212.194:34515
server_dump_info_path | /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.14.212.195:42281
server_dump_info_path | /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.14.212.196:43697
server_dump_info_path | /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.14.212.197:41667
server_dump_info_path | /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.14.212.198:34487
server_dump_info_path | /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.14.212.199:45747
server_dump_info_path | /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.14.212.200:41413
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.14.212.212:37613 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 01:53:22.488731 15187 log_verifier.cc:126] Checking tablet 037045b61419472089771a8c6a057d21
I20240502 01:53:22.499882 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 037045b61419472089771a8c6a057d21
I20240502 01:53:22.500000 15187 log_verifier.cc:126] Checking tablet 149a3a4584fe4c82a59ca4bc28ca80dd
I20240502 01:53:22.510048 15187 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 149a3a4584fe4c82a59ca4bc28ca80dd
I20240502 01:53:22.510190 15187 log_verifier.cc:126] Checking tablet 1e8092e428084a56bbfaaca963877631
I20240502 01:53:22.520305 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 1e8092e428084a56bbfaaca963877631
I20240502 01:53:22.520414 15187 log_verifier.cc:126] Checking tablet 2bfc0209bd434fa9a56b45d7a9b73a12
I20240502 01:53:22.530616 15187 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2bfc0209bd434fa9a56b45d7a9b73a12
I20240502 01:53:22.530714 15187 log_verifier.cc:126] Checking tablet 3fed2519c9b34871b27e5148e18b0673
I20240502 01:53:22.541157 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 3fed2519c9b34871b27e5148e18b0673
I20240502 01:53:22.541281 15187 log_verifier.cc:126] Checking tablet 4d4eaec689d345b4a812a448e5d359e4
I20240502 01:53:22.551079 15187 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4d4eaec689d345b4a812a448e5d359e4
I20240502 01:53:22.551182 15187 log_verifier.cc:126] Checking tablet 97ffbdb149fb42c892ed094ce60f7166
I20240502 01:53:22.560984 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 97ffbdb149fb42c892ed094ce60f7166
I20240502 01:53:22.561086 15187 log_verifier.cc:126] Checking tablet a487f57191fe4eb998ad8f5f86f4050b
I20240502 01:53:22.566314 15187 log_verifier.cc:177] Verified matching terms for 2 ops in tablet a487f57191fe4eb998ad8f5f86f4050b
I20240502 01:53:22.566409 15187 log_verifier.cc:126] Checking tablet b1636e2cd56f40359df64d2ac4c9d806
I20240502 01:53:22.576639 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b1636e2cd56f40359df64d2ac4c9d806
I20240502 01:53:22.576747 15187 log_verifier.cc:126] Checking tablet b477b96739354b6ab43bf4ee230f4559
I20240502 01:53:22.587191 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b477b96739354b6ab43bf4ee230f4559
I20240502 01:53:22.587311 15187 log_verifier.cc:126] Checking tablet ba6e000d044241edbb347c80dca80b58
I20240502 01:53:22.601527 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet ba6e000d044241edbb347c80dca80b58
I20240502 01:53:22.601641 15187 log_verifier.cc:126] Checking tablet bc1d0f4745df4203848a1f4820f9b499
I20240502 01:53:22.611688 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet bc1d0f4745df4203848a1f4820f9b499
I20240502 01:53:22.611783 15187 log_verifier.cc:126] Checking tablet bf441bb1e6064b56b745ffe8d02a3983
I20240502 01:53:22.625718 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet bf441bb1e6064b56b745ffe8d02a3983
I20240502 01:53:22.625842 15187 log_verifier.cc:126] Checking tablet c130cb706b004c2088925fba192a8c90
I20240502 01:53:22.636126 15187 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c130cb706b004c2088925fba192a8c90
I20240502 01:53:22.636238 15187 log_verifier.cc:126] Checking tablet d94d0492ef1d4603b77441d09ad3c6dc
I20240502 01:53:22.646421 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d94d0492ef1d4603b77441d09ad3c6dc
I20240502 01:53:22.646536 15187 log_verifier.cc:126] Checking tablet e4a22d109d8044adaeb364037e9c137e
I20240502 01:53:22.657301 15187 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e4a22d109d8044adaeb364037e9c137e
I20240502 01:53:22.658524 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 17146
I20240502 01:53:22.659912 17269 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:22.874471 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 17146
I20240502 01:53:22.910109 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 17279
I20240502 01:53:22.911172 17402 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:23.118854 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 17279
I20240502 01:53:23.152971 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 18391
I20240502 01:53:23.159459 18516 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:23.357046 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 18391
I20240502 01:53:23.388033 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 18528
I20240502 01:53:23.394214 18659 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:23.582198 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 18528
W20240502 01:53:23.612249 17072 connection.cc:535] server connection from 127.14.212.196:52109 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:23.612566 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 18668
I20240502 01:53:23.618688 18792 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:23.809154 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 18668
I20240502 01:53:23.840576 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 18800
I20240502 01:53:23.846832 18923 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:24.035459 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 18800
I20240502 01:53:24.066074 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 18932
I20240502 01:53:24.072517 19055 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:24.258486 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 18932
I20240502 01:53:24.289479 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 20950
I20240502 01:53:24.295980 21074 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:24.411828 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 20950
W20240502 01:53:24.433228 17073 connection.cc:535] server connection from 127.14.212.200:46479 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:24.433418 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 17056
I20240502 01:53:24.434495 17117 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:24.677788 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 17056
2024-05-02T01:53:24Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/0 (44620 ms)
[----------] 1 test from RebalanceParamTest (44620 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0
I20240502 01:53:24.762207 15187 test_util.cc:268] Using random seed: -564310458
I20240502 01:53:24.764142 15187 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:53:24.764200 15187 ts_itest-base.cc:113] --------------
I20240502 01:53:24.764243 15187 ts_itest-base.cc:114] 8 tablet servers
I20240502 01:53:24.764278 15187 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:53:24.764320 15187 ts_itest-base.cc:116] --------------
2024-05-02T01:53:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:53:24Z Disabled control of system clock
I20240502 01:53:24.794075 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.254:33157
--webserver_interface=127.14.212.254
--webserver_port=0
--builtin_ntp_servers=127.14.212.212:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.254:33157
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 01:53:25.143203 21684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:25.143528 21684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:25.143633 21684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:25.151440 21684 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:25.151585 21684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:53:25.151677 21684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:25.151769 21684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:53:25.151839 21684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:53:25.161435 21684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.14.212.254:33157
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.254:33157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.212.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:25.162917 21684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:25.164839 21684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:25.176965 21690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:25.177526 21689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:25.178129 21692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:25.178956 21684 server_base.cc:1034] running on GCE node
I20240502 01:53:25.179705 21684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:25.181238 21684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:25.182442 21684 hybrid_clock.cc:648] HybridClock initialized: now 1714614805182385 us; error 52 us; skew 500 ppm
I20240502 01:53:25.182832 21684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:25.185276 21684 webserver.cc:460] Webserver started at http://127.14.212.254:39429/ using document root <none> and password file <none>
I20240502 01:53:25.186029 21684 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:25.186147 21684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:25.186468 21684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:25.189299 21684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f9e4d9a6ca2a4b698f272d98462e62bb"
format_stamp: "Formatted at 2024-05-02 01:53:25 on dist-test-slave-mk1r"
I20240502 01:53:25.190130 21684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f9e4d9a6ca2a4b698f272d98462e62bb"
format_stamp: "Formatted at 2024-05-02 01:53:25 on dist-test-slave-mk1r"
I20240502 01:53:25.195791 21684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 01:53:25.199477 21698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:25.200906 21684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:25.201093 21684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f9e4d9a6ca2a4b698f272d98462e62bb"
format_stamp: "Formatted at 2024-05-02 01:53:25 on dist-test-slave-mk1r"
I20240502 01:53:25.201318 21684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:25.217706 21684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:25.218618 21684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:25.218940 21684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:25.238116 21684 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.254:33157
I20240502 01:53:25.238135 21750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.254:33157 every 8 connection(s)
I20240502 01:53:25.239519 21684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:53:25.243242 21751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:25.245268 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 21684
I20240502 01:53:25.245492 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:53:25.254508 21751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Bootstrap starting.
I20240502 01:53:25.258001 21751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:25.259395 21751 log.cc:824] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:25.262660 21751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: No bootstrap required, opened a new log
I20240502 01:53:25.270751 21751 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e4d9a6ca2a4b698f272d98462e62bb" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 33157 } }
I20240502 01:53:25.271248 21751 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:25.271392 21751 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e4d9a6ca2a4b698f272d98462e62bb, State: Initialized, Role: FOLLOWER
I20240502 01:53:25.272051 21751 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e4d9a6ca2a4b698f272d98462e62bb" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 33157 } }
I20240502 01:53:25.272269 21751 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:53:25.272379 21751 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:53:25.272509 21751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:25.274763 21751 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e4d9a6ca2a4b698f272d98462e62bb" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 33157 } }
I20240502 01:53:25.275386 21751 leader_election.cc:302] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9e4d9a6ca2a4b698f272d98462e62bb; no voters:
I20240502 01:53:25.276082 21751 leader_election.cc:288] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:53:25.276235 21756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:25.276942 21756 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb [term 1 LEADER]: Becoming Leader. State: Replica: f9e4d9a6ca2a4b698f272d98462e62bb, State: Running, Role: LEADER
I20240502 01:53:25.277525 21756 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e4d9a6ca2a4b698f272d98462e62bb" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 33157 } }
I20240502 01:53:25.278147 21751 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:53:25.280202 21758 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9e4d9a6ca2a4b698f272d98462e62bb. Latest consensus state: current_term: 1 leader_uuid: "f9e4d9a6ca2a4b698f272d98462e62bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e4d9a6ca2a4b698f272d98462e62bb" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 33157 } } }
I20240502 01:53:25.280493 21758 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:25.280959 21757 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9e4d9a6ca2a4b698f272d98462e62bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e4d9a6ca2a4b698f272d98462e62bb" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 33157 } } }
I20240502 01:53:25.281291 21757 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:25.281481 21762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:53:25.291949 21762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:53:25.301095 21762 catalog_manager.cc:1347] Generated new cluster ID: f7afede68a194eaeb50e008ec7fa0d7f
I20240502 01:53:25.301237 21762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:53:25.319526 21762 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:53:25.320793 21762 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:53:25.341344 21762 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Generated new TSK 0
I20240502 01:53:25.342213 21762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:53:25.351372 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.193:0
--local_ip_for_outbound_sockets=127.14.212.193
--webserver_interface=127.14.212.193
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33157
--builtin_ntp_servers=127.14.212.212:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:53:25.699234 21775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:25.699542 21775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:25.699690 21775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:25.707583 21775 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:25.707718 21775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:25.707860 21775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.193
I20240502 01:53:25.717330 21775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.212.193
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33157
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.193
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:25.718714 21775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:25.720465 21775 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:25.733407 21781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:25.733942 21783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:25.734618 21780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:25.735343 21775 server_base.cc:1034] running on GCE node
I20240502 01:53:25.736055 21775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:25.737111 21775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:25.738343 21775 hybrid_clock.cc:648] HybridClock initialized: now 1714614805738278 us; error 55 us; skew 500 ppm
I20240502 01:53:25.738732 21775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:25.741225 21775 webserver.cc:460] Webserver started at http://127.14.212.193:43963/ using document root <none> and password file <none>
I20240502 01:53:25.742026 21775 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:25.742146 21775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:25.742480 21775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:25.745292 21775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "759257ea7bef415ca698e60d82199d77"
format_stamp: "Formatted at 2024-05-02 01:53:25 on dist-test-slave-mk1r"
I20240502 01:53:25.746073 21775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "759257ea7bef415ca698e60d82199d77"
format_stamp: "Formatted at 2024-05-02 01:53:25 on dist-test-slave-mk1r"
I20240502 01:53:25.751309 21775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 01:53:25.754822 21789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:25.756157 21775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:25.756312 21775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "759257ea7bef415ca698e60d82199d77"
format_stamp: "Formatted at 2024-05-02 01:53:25 on dist-test-slave-mk1r"
I20240502 01:53:25.756528 21775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:25.778155 21775 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:25.779031 21775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:25.779340 21775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:25.780483 21775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:25.782138 21775 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:25.782240 21775 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:25.782359 21775 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:25.782433 21775 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:25.816934 21775 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.193:33785
I20240502 01:53:25.816998 21902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.193:33785 every 8 connection(s)
I20240502 01:53:25.818324 21775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:53:25.826442 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 21775
I20240502 01:53:25.826663 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:53:25.830215 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.194:0
--local_ip_for_outbound_sockets=127.14.212.194
--webserver_interface=127.14.212.194
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33157
--builtin_ntp_servers=127.14.212.212:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:25.831799 21903 heartbeater.cc:347] Connected to a master server at 127.14.212.254:33157
I20240502 01:53:25.832123 21903 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:25.832962 21903 heartbeater.cc:511] Master 127.14.212.254:33157 requested a full tablet report, sending...
I20240502 01:53:25.835119 21715 ts_manager.cc:194] Registered new tserver with Master: 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:25.836673 21715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.193:38923
W20240502 01:53:26.166493 21907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:26.166800 21907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:26.166887 21907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:26.174381 21907 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:26.174510 21907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:26.174652 21907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.194
I20240502 01:53:26.184250 21907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.212.194
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33157
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.194
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:26.185629 21907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:26.187469 21907 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:26.199570 21913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:26.199683 21915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:26.199875 21912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:26.200452 21907 server_base.cc:1034] running on GCE node
I20240502 01:53:26.201169 21907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:26.202145 21907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:26.203404 21907 hybrid_clock.cc:648] HybridClock initialized: now 1714614806203358 us; error 37 us; skew 500 ppm
I20240502 01:53:26.203758 21907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:26.206171 21907 webserver.cc:460] Webserver started at http://127.14.212.194:36423/ using document root <none> and password file <none>
I20240502 01:53:26.206969 21907 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:26.207098 21907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:26.207424 21907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:26.210389 21907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "06543b05a16f42099bc86db2a0284412"
format_stamp: "Formatted at 2024-05-02 01:53:26 on dist-test-slave-mk1r"
I20240502 01:53:26.211201 21907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "06543b05a16f42099bc86db2a0284412"
format_stamp: "Formatted at 2024-05-02 01:53:26 on dist-test-slave-mk1r"
I20240502 01:53:26.216500 21907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 01:53:26.220260 21921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:26.221719 21907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:53:26.221886 21907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "06543b05a16f42099bc86db2a0284412"
format_stamp: "Formatted at 2024-05-02 01:53:26 on dist-test-slave-mk1r"
I20240502 01:53:26.222095 21907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:26.243214 21907 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:26.244082 21907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:26.244357 21907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:26.245422 21907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:26.247107 21907 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:26.247206 21907 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:26.247327 21907 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:26.247406 21907 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:26.282435 21907 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.194:44411
I20240502 01:53:26.282503 22034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.194:44411 every 8 connection(s)
I20240502 01:53:26.283864 21907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:53:26.294018 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 21907
I20240502 01:53:26.294239 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:53:26.296763 22035 heartbeater.cc:347] Connected to a master server at 127.14.212.254:33157
I20240502 01:53:26.297147 22035 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:26.297952 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.195:0
--local_ip_for_outbound_sockets=127.14.212.195
--webserver_interface=127.14.212.195
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33157
--builtin_ntp_servers=127.14.212.212:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:26.298173 22035 heartbeater.cc:511] Master 127.14.212.254:33157 requested a full tablet report, sending...
I20240502 01:53:26.299834 21715 ts_manager.cc:194] Registered new tserver with Master: 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:26.300655 21715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.194:48539
W20240502 01:53:26.645217 22038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:26.645555 22038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:26.645707 22038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:26.653329 22038 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:26.653491 22038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:26.653659 22038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.195
I20240502 01:53:26.662988 22038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.212.195
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33157
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.195
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:26.664340 22038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:26.666134 22038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:26.677811 22043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:26.677932 22044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:26.678417 22046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:26.678982 22038 server_base.cc:1034] running on GCE node
I20240502 01:53:26.679623 22038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:26.680554 22038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:26.681764 22038 hybrid_clock.cc:648] HybridClock initialized: now 1714614806681716 us; error 38 us; skew 500 ppm
I20240502 01:53:26.682108 22038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:26.684340 22038 webserver.cc:460] Webserver started at http://127.14.212.195:38683/ using document root <none> and password file <none>
I20240502 01:53:26.685132 22038 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:26.685246 22038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:26.685555 22038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:26.688210 22038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c64840657cdf4b93980bd34120c94d9a"
format_stamp: "Formatted at 2024-05-02 01:53:26 on dist-test-slave-mk1r"
I20240502 01:53:26.688994 22038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c64840657cdf4b93980bd34120c94d9a"
format_stamp: "Formatted at 2024-05-02 01:53:26 on dist-test-slave-mk1r"
I20240502 01:53:26.694161 22038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240502 01:53:26.697539 22052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:26.698837 22038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:53:26.698998 22038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c64840657cdf4b93980bd34120c94d9a"
format_stamp: "Formatted at 2024-05-02 01:53:26 on dist-test-slave-mk1r"
I20240502 01:53:26.699199 22038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:26.731693 22038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:26.732605 22038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:26.732941 22038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:26.733945 22038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:26.735531 22038 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:26.735637 22038 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:26.735778 22038 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:26.735891 22038 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:26.768399 22038 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.195:38879
I20240502 01:53:26.768467 22165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.195:38879 every 8 connection(s)
I20240502 01:53:26.769769 22038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:53:26.771936 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 22038
I20240502 01:53:26.772166 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:53:26.775616 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.196:0
--local_ip_for_outbound_sockets=127.14.212.196
--webserver_interface=127.14.212.196
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33157
--builtin_ntp_servers=127.14.212.212:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:26.783386 22166 heartbeater.cc:347] Connected to a master server at 127.14.212.254:33157
I20240502 01:53:26.783733 22166 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:26.784727 22166 heartbeater.cc:511] Master 127.14.212.254:33157 requested a full tablet report, sending...
I20240502 01:53:26.786419 21715 ts_manager.cc:194] Registered new tserver with Master: c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:26.787642 21715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.195:46943
I20240502 01:53:26.840207 21903 heartbeater.cc:503] Master 127.14.212.254:33157 was elected leader, sending a full tablet report...
W20240502 01:53:27.110790 22169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:27.111094 22169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:27.111184 22169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:27.118654 22169 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:27.118793 22169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:27.118919 22169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.196
I20240502 01:53:27.128190 22169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.212.196
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33157
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.196
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:27.129560 22169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:27.131227 22169 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:27.143029 22177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:27.143146 22175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:27.143038 22174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:27.143911 22169 server_base.cc:1034] running on GCE node
I20240502 01:53:27.144553 22169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:27.145552 22169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:27.146777 22169 hybrid_clock.cc:648] HybridClock initialized: now 1714614807146721 us; error 42 us; skew 500 ppm
I20240502 01:53:27.147133 22169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:27.149536 22169 webserver.cc:460] Webserver started at http://127.14.212.196:43123/ using document root <none> and password file <none>
I20240502 01:53:27.150303 22169 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:27.150419 22169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:27.150743 22169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:27.153482 22169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "c2a28baa7ee2469294d94b7fe5455cff"
format_stamp: "Formatted at 2024-05-02 01:53:27 on dist-test-slave-mk1r"
I20240502 01:53:27.154243 22169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "c2a28baa7ee2469294d94b7fe5455cff"
format_stamp: "Formatted at 2024-05-02 01:53:27 on dist-test-slave-mk1r"
I20240502 01:53:27.159590 22169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 01:53:27.163317 22183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:27.164826 22169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:27.165015 22169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c2a28baa7ee2469294d94b7fe5455cff"
format_stamp: "Formatted at 2024-05-02 01:53:27 on dist-test-slave-mk1r"
I20240502 01:53:27.165206 22169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:27.196051 22169 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:27.197088 22169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:27.197382 22169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:27.198374 22169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:27.199957 22169 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:27.200047 22169 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:27.200179 22169 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:27.200253 22169 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:27.233536 22169 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.196:43093
I20240502 01:53:27.233597 22296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.196:43093 every 8 connection(s)
I20240502 01:53:27.234951 22169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:53:27.240067 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 22169
I20240502 01:53:27.240269 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:53:27.244004 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.197:0
--local_ip_for_outbound_sockets=127.14.212.197
--webserver_interface=127.14.212.197
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33157
--builtin_ntp_servers=127.14.212.212:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:27.250337 22297 heartbeater.cc:347] Connected to a master server at 127.14.212.254:33157
I20240502 01:53:27.250749 22297 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:27.251832 22297 heartbeater.cc:511] Master 127.14.212.254:33157 requested a full tablet report, sending...
I20240502 01:53:27.253445 21715 ts_manager.cc:194] Registered new tserver with Master: c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:27.254316 21715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.196:40573
I20240502 01:53:27.303817 22035 heartbeater.cc:503] Master 127.14.212.254:33157 was elected leader, sending a full tablet report...
W20240502 01:53:27.588420 22300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:27.588762 22300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:27.588933 22300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:27.596745 22300 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:27.596971 22300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:27.597147 22300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.197
I20240502 01:53:27.606984 22300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.212.197
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33157
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.197
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:27.608422 22300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:27.610286 22300 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:27.623522 22308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:27.623466 22306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:27.623567 22305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:27.624084 22300 server_base.cc:1034] running on GCE node
I20240502 01:53:27.624830 22300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:27.625941 22300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:27.627187 22300 hybrid_clock.cc:648] HybridClock initialized: now 1714614807627138 us; error 41 us; skew 500 ppm
I20240502 01:53:27.627601 22300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:27.630100 22300 webserver.cc:460] Webserver started at http://127.14.212.197:35037/ using document root <none> and password file <none>
I20240502 01:53:27.630949 22300 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:27.631085 22300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:27.631414 22300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:27.634300 22300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
format_stamp: "Formatted at 2024-05-02 01:53:27 on dist-test-slave-mk1r"
I20240502 01:53:27.635166 22300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
format_stamp: "Formatted at 2024-05-02 01:53:27 on dist-test-slave-mk1r"
I20240502 01:53:27.640926 22300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20240502 01:53:27.644801 22314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:27.646389 22300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240502 01:53:27.646589 22300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
format_stamp: "Formatted at 2024-05-02 01:53:27 on dist-test-slave-mk1r"
I20240502 01:53:27.646826 22300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:27.672168 22300 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:27.673106 22300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:27.673363 22300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:27.674315 22300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:27.675979 22300 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:27.676070 22300 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:27.676158 22300 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:27.676215 22300 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:27.707818 22300 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.197:36677
I20240502 01:53:27.707878 22426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.197:36677 every 8 connection(s)
I20240502 01:53:27.709282 22300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:53:27.719777 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 22300
I20240502 01:53:27.720008 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:53:27.722663 22427 heartbeater.cc:347] Connected to a master server at 127.14.212.254:33157
I20240502 01:53:27.722987 22427 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:27.723436 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.198:0
--local_ip_for_outbound_sockets=127.14.212.198
--webserver_interface=127.14.212.198
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33157
--builtin_ntp_servers=127.14.212.212:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:27.723838 22427 heartbeater.cc:511] Master 127.14.212.254:33157 requested a full tablet report, sending...
I20240502 01:53:27.725634 21715 ts_manager.cc:194] Registered new tserver with Master: 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:53:27.726796 21715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.197:41143
I20240502 01:53:27.790799 22166 heartbeater.cc:503] Master 127.14.212.254:33157 was elected leader, sending a full tablet report...
W20240502 01:53:28.070624 22430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:28.070955 22430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:28.071098 22430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:28.078918 22430 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:28.079069 22430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:28.079259 22430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.198
I20240502 01:53:28.088676 22430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.14.212.198
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33157
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.198
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:28.090173 22430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:28.091946 22430 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:28.104293 22438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:28.104384 22436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:28.104743 22435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:28.105541 22430 server_base.cc:1034] running on GCE node
I20240502 01:53:28.106130 22430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:28.107105 22430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:28.108317 22430 hybrid_clock.cc:648] HybridClock initialized: now 1714614808108257 us; error 56 us; skew 500 ppm
I20240502 01:53:28.108659 22430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:28.111017 22430 webserver.cc:460] Webserver started at http://127.14.212.198:41119/ using document root <none> and password file <none>
I20240502 01:53:28.111804 22430 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:28.111917 22430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:28.112233 22430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:28.114944 22430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "28157bc3e678428fb5f93c077d528513"
format_stamp: "Formatted at 2024-05-02 01:53:28 on dist-test-slave-mk1r"
I20240502 01:53:28.115707 22430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "28157bc3e678428fb5f93c077d528513"
format_stamp: "Formatted at 2024-05-02 01:53:28 on dist-test-slave-mk1r"
I20240502 01:53:28.121168 22430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240502 01:53:28.124570 22444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:28.125944 22430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:53:28.126101 22430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "28157bc3e678428fb5f93c077d528513"
format_stamp: "Formatted at 2024-05-02 01:53:28 on dist-test-slave-mk1r"
I20240502 01:53:28.126299 22430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:28.146706 22430 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:28.147639 22430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:28.147914 22430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:28.148932 22430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:28.150508 22430 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:28.150622 22430 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:28.150744 22430 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:28.150817 22430 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:28.183897 22430 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.198:42375
I20240502 01:53:28.183951 22556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.198:42375 every 8 connection(s)
I20240502 01:53:28.185285 22430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:53:28.188462 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 22430
I20240502 01:53:28.188659 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:53:28.192207 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.199:0
--local_ip_for_outbound_sockets=127.14.212.199
--webserver_interface=127.14.212.199
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33157
--builtin_ntp_servers=127.14.212.212:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:28.199491 22557 heartbeater.cc:347] Connected to a master server at 127.14.212.254:33157
I20240502 01:53:28.199811 22557 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:28.200573 22557 heartbeater.cc:511] Master 127.14.212.254:33157 requested a full tablet report, sending...
I20240502 01:53:28.202160 21715 ts_manager.cc:194] Registered new tserver with Master: 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:53:28.202966 21715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.198:59605
I20240502 01:53:28.257433 22297 heartbeater.cc:503] Master 127.14.212.254:33157 was elected leader, sending a full tablet report...
W20240502 01:53:28.540756 22560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:28.541128 22560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:28.541280 22560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:28.549031 22560 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:28.549206 22560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:28.549382 22560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.199
I20240502 01:53:28.559176 22560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.14.212.199
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33157
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.199
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:28.560640 22560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:28.562590 22560 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:28.576126 22565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:28.576288 22566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:28.576493 22568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:28.577021 22560 server_base.cc:1034] running on GCE node
I20240502 01:53:28.577661 22560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:28.578789 22560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:28.579998 22560 hybrid_clock.cc:648] HybridClock initialized: now 1714614808579957 us; error 38 us; skew 500 ppm
I20240502 01:53:28.580327 22560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:28.582710 22560 webserver.cc:460] Webserver started at http://127.14.212.199:41269/ using document root <none> and password file <none>
I20240502 01:53:28.583478 22560 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:28.583609 22560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:28.583921 22560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:28.587364 22560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "d7449bb1df214bf4a5f93e4631312f08"
format_stamp: "Formatted at 2024-05-02 01:53:28 on dist-test-slave-mk1r"
I20240502 01:53:28.588272 22560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "d7449bb1df214bf4a5f93e4631312f08"
format_stamp: "Formatted at 2024-05-02 01:53:28 on dist-test-slave-mk1r"
I20240502 01:53:28.595110 22560 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20240502 01:53:28.598767 22574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:28.600260 22560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 01:53:28.600426 22560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "d7449bb1df214bf4a5f93e4631312f08"
format_stamp: "Formatted at 2024-05-02 01:53:28 on dist-test-slave-mk1r"
I20240502 01:53:28.600657 22560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:28.620728 22560 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:28.621673 22560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:28.621956 22560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:28.622961 22560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:28.624559 22560 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:28.624670 22560 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:28.624790 22560 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:28.624866 22560 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:28.658906 22560 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.199:35737
I20240502 01:53:28.658951 22686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.199:35737 every 8 connection(s)
I20240502 01:53:28.660415 22560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:53:28.668022 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 22560
I20240502 01:53:28.668282 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 01:53:28.671872 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.200:0
--local_ip_for_outbound_sockets=127.14.212.200
--webserver_interface=127.14.212.200
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33157
--builtin_ntp_servers=127.14.212.212:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:28.674357 22687 heartbeater.cc:347] Connected to a master server at 127.14.212.254:33157
I20240502 01:53:28.674741 22687 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:28.675760 22687 heartbeater.cc:511] Master 127.14.212.254:33157 requested a full tablet report, sending...
I20240502 01:53:28.677414 21715 ts_manager.cc:194] Registered new tserver with Master: d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:53:28.678287 21715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.199:50889
I20240502 01:53:28.729497 22427 heartbeater.cc:503] Master 127.14.212.254:33157 was elected leader, sending a full tablet report...
W20240502 01:53:29.019547 22690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:29.019912 22690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:29.020077 22690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:29.028184 22690 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:29.028352 22690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:29.028520 22690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.200
I20240502 01:53:29.038622 22690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.14.212.200
--webserver_port=0
--tserver_master_addrs=127.14.212.254:33157
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.200
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:29.040050 22690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:29.041859 22690 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:29.053767 22695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:29.053903 22698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:29.054095 22696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:29.054771 22690 server_base.cc:1034] running on GCE node
I20240502 01:53:29.055405 22690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:29.056375 22690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:29.057598 22690 hybrid_clock.cc:648] HybridClock initialized: now 1714614809057527 us; error 51 us; skew 500 ppm
I20240502 01:53:29.057929 22690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:29.060213 22690 webserver.cc:460] Webserver started at http://127.14.212.200:34041/ using document root <none> and password file <none>
I20240502 01:53:29.061024 22690 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:29.061147 22690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:29.061480 22690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:29.064222 22690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "41c70f373c9d49a49891b03a12c6cb2b"
format_stamp: "Formatted at 2024-05-02 01:53:29 on dist-test-slave-mk1r"
I20240502 01:53:29.065029 22690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "41c70f373c9d49a49891b03a12c6cb2b"
format_stamp: "Formatted at 2024-05-02 01:53:29 on dist-test-slave-mk1r"
I20240502 01:53:29.070644 22690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240502 01:53:29.074245 22704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:29.075646 22690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240502 01:53:29.075814 22690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "41c70f373c9d49a49891b03a12c6cb2b"
format_stamp: "Formatted at 2024-05-02 01:53:29 on dist-test-slave-mk1r"
I20240502 01:53:29.076020 22690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:29.101773 22690 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:29.102691 22690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:29.102972 22690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:29.104081 22690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:29.105819 22690 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:29.105927 22690 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:29.106036 22690 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:29.106114 22690 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:29.139263 22690 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.200:37097
I20240502 01:53:29.139331 22816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.200:37097 every 8 connection(s)
I20240502 01:53:29.140578 22690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:53:29.146873 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 22690
I20240502 01:53:29.147207 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 01:53:29.153851 22817 heartbeater.cc:347] Connected to a master server at 127.14.212.254:33157
I20240502 01:53:29.154143 22817 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:29.154886 22817 heartbeater.cc:511] Master 127.14.212.254:33157 requested a full tablet report, sending...
I20240502 01:53:29.156145 21715 ts_manager.cc:194] Registered new tserver with Master: 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:53:29.156977 21715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.200:42099
I20240502 01:53:29.164130 15187 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:53:29.181247 15187 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:53:29.181408 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 22300
I20240502 01:53:29.190644 22422 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:29.206204 22557 heartbeater.cc:503] Master 127.14.212.254:33157 was elected leader, sending a full tablet report...
I20240502 01:53:29.275048 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 22300
I20240502 01:53:29.293476 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 22430
I20240502 01:53:29.302325 22552 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:29.389931 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 22430
I20240502 01:53:29.407505 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 22560
I20240502 01:53:29.415769 22682 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:29.496448 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 22560
I20240502 01:53:29.513653 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 22690
I20240502 01:53:29.522068 22812 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:29.604054 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 22690
I20240502 01:53:33.373193 21715 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:33.428501 22230 tablet_service.cc:1467] Processing CreateTablet for tablet cddc036193de43c9b7569f664f3e4d7c (DEFAULT_TABLE table=rebalance_test_table_0 [id=14af009e70d54323b1c55d5cc89d84d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.429662 22231 tablet_service.cc:1467] Processing CreateTablet for tablet b02ca6aec3774fe5a1aff6ee307cbad5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=14af009e70d54323b1c55d5cc89d84d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.431074 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cddc036193de43c9b7569f664f3e4d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.431634 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b02ca6aec3774fe5a1aff6ee307cbad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.431582 21837 tablet_service.cc:1467] Processing CreateTablet for tablet b02ca6aec3774fe5a1aff6ee307cbad5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=14af009e70d54323b1c55d5cc89d84d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.431998 22099 tablet_service.cc:1467] Processing CreateTablet for tablet cddc036193de43c9b7569f664f3e4d7c (DEFAULT_TABLE table=rebalance_test_table_0 [id=14af009e70d54323b1c55d5cc89d84d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.433441 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cddc036193de43c9b7569f664f3e4d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.433451 21835 tablet_service.cc:1467] Processing CreateTablet for tablet cddc036193de43c9b7569f664f3e4d7c (DEFAULT_TABLE table=rebalance_test_table_0 [id=14af009e70d54323b1c55d5cc89d84d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.434090 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 95fa4e82b6a44b5bbc0f689a318090f7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=14af009e70d54323b1c55d5cc89d84d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.434589 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 95fa4e82b6a44b5bbc0f689a318090f7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=14af009e70d54323b1c55d5cc89d84d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.435676 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95fa4e82b6a44b5bbc0f689a318090f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.435889 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b02ca6aec3774fe5a1aff6ee307cbad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.436342 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cddc036193de43c9b7569f664f3e4d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.437153 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95fa4e82b6a44b5bbc0f689a318090f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.451715 21969 tablet_service.cc:1467] Processing CreateTablet for tablet b02ca6aec3774fe5a1aff6ee307cbad5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=14af009e70d54323b1c55d5cc89d84d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.453843 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b02ca6aec3774fe5a1aff6ee307cbad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.457891 21968 tablet_service.cc:1467] Processing CreateTablet for tablet 95fa4e82b6a44b5bbc0f689a318090f7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=14af009e70d54323b1c55d5cc89d84d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.458539 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95fa4e82b6a44b5bbc0f689a318090f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.471027 22848 tablet_bootstrap.cc:492] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:33.476545 22848 tablet_bootstrap.cc:654] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.470319 22847 tablet_bootstrap.cc:492] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:33.478536 22849 tablet_bootstrap.cc:492] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:33.478533 22848 log.cc:824] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:33.479946 22846 tablet_bootstrap.cc:492] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:33.483173 22846 tablet_bootstrap.cc:654] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.483183 22847 tablet_bootstrap.cc:654] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.483253 22849 tablet_bootstrap.cc:654] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.484810 22849 log.cc:824] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:33.485059 22846 log.cc:824] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:33.486287 22847 log.cc:824] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:33.488416 22848 tablet_bootstrap.cc:492] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:33.488756 22848 ts_tablet_manager.cc:1367] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20240502 01:53:33.489243 22849 tablet_bootstrap.cc:492] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:33.489488 22846 tablet_bootstrap.cc:492] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:33.489537 22849 ts_tablet_manager.cc:1367] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20240502 01:53:33.489879 22846 ts_tablet_manager.cc:1367] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240502 01:53:33.490065 22847 tablet_bootstrap.cc:492] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:33.490350 22847 ts_tablet_manager.cc:1367] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.000s
I20240502 01:53:33.498536 22848 raft_consensus.cc:348] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:33.499075 22848 raft_consensus.cc:374] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.499197 22848 raft_consensus.cc:729] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.499270 22849 raft_consensus.cc:348] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:33.499722 22849 raft_consensus.cc:374] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.499413 22847 raft_consensus.cc:348] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:33.499833 22849 raft_consensus.cc:729] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.499878 22847 raft_consensus.cc:374] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.499985 22847 raft_consensus.cc:729] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.500133 22848 consensus_queue.cc:260] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:33.500497 22849 consensus_queue.cc:260] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:33.500929 22847 consensus_queue.cc:260] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:33.501557 22846 raft_consensus.cc:348] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:33.502204 22846 raft_consensus.cc:374] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.502396 22846 raft_consensus.cc:729] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.503229 22846 consensus_queue.cc:260] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:33.509114 22848 ts_tablet_manager.cc:1398] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.020s user 0.010s sys 0.011s
I20240502 01:53:33.510841 22849 ts_tablet_manager.cc:1398] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.021s user 0.018s sys 0.002s
I20240502 01:53:33.511396 22848 tablet_bootstrap.cc:492] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:33.513298 22848 tablet_bootstrap.cc:654] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.516094 22847 ts_tablet_manager.cc:1398] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.026s user 0.002s sys 0.023s
I20240502 01:53:33.516685 22849 tablet_bootstrap.cc:492] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:33.516824 22847 tablet_bootstrap.cc:492] T cddc036193de43c9b7569f664f3e4d7c P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:33.518595 22849 tablet_bootstrap.cc:654] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.523603 22846 ts_tablet_manager.cc:1398] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.034s user 0.017s sys 0.009s
I20240502 01:53:33.524274 22846 tablet_bootstrap.cc:492] T cddc036193de43c9b7569f664f3e4d7c P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:33.524488 22847 tablet_bootstrap.cc:654] T cddc036193de43c9b7569f664f3e4d7c P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.526130 22846 tablet_bootstrap.cc:654] T cddc036193de43c9b7569f664f3e4d7c P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
W20240502 01:53:33.527271 22167 tablet.cc:2369] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:33.528983 22848 tablet_bootstrap.cc:492] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:33.529209 22848 ts_tablet_manager.cc:1367] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240502 01:53:33.530186 22846 tablet_bootstrap.cc:492] T cddc036193de43c9b7569f664f3e4d7c P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:33.530346 22846 ts_tablet_manager.cc:1367] T cddc036193de43c9b7569f664f3e4d7c P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240502 01:53:33.530359 22849 tablet_bootstrap.cc:492] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:33.530525 22849 ts_tablet_manager.cc:1367] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20240502 01:53:33.530500 22848 raft_consensus.cc:348] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:33.530748 22848 raft_consensus.cc:374] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.530862 22848 raft_consensus.cc:729] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.531215 22848 consensus_queue.cc:260] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:33.531358 22846 raft_consensus.cc:348] T cddc036193de43c9b7569f664f3e4d7c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:33.531584 22846 raft_consensus.cc:374] T cddc036193de43c9b7569f664f3e4d7c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.531692 22847 tablet_bootstrap.cc:492] T cddc036193de43c9b7569f664f3e4d7c P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:33.531690 22846 raft_consensus.cc:729] T cddc036193de43c9b7569f664f3e4d7c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.531913 22847 ts_tablet_manager.cc:1367] T cddc036193de43c9b7569f664f3e4d7c P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240502 01:53:33.531756 22849 raft_consensus.cc:348] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:33.531934 22846 consensus_queue.cc:260] T cddc036193de43c9b7569f664f3e4d7c P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:33.532090 22849 raft_consensus.cc:374] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.532204 22849 raft_consensus.cc:729] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.532503 22846 ts_tablet_manager.cc:1398] T cddc036193de43c9b7569f664f3e4d7c P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:33.532544 22849 consensus_queue.cc:260] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:33.533053 22848 ts_tablet_manager.cc:1398] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:33.533214 22849 ts_tablet_manager.cc:1398] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:33.533360 22848 tablet_bootstrap.cc:492] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:33.535228 22848 tablet_bootstrap.cc:654] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.537616 22847 raft_consensus.cc:348] T cddc036193de43c9b7569f664f3e4d7c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:33.537817 22847 raft_consensus.cc:374] T cddc036193de43c9b7569f664f3e4d7c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.537930 22847 raft_consensus.cc:729] T cddc036193de43c9b7569f664f3e4d7c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.538199 22847 consensus_queue.cc:260] T cddc036193de43c9b7569f664f3e4d7c P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:33.538764 22848 tablet_bootstrap.cc:492] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:33.538823 22847 ts_tablet_manager.cc:1398] T cddc036193de43c9b7569f664f3e4d7c P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 01:53:33.538957 22848 ts_tablet_manager.cc:1367] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:53:33.539614 22848 raft_consensus.cc:348] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:33.539800 22848 raft_consensus.cc:374] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.539877 22848 raft_consensus.cc:729] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.540141 22848 consensus_queue.cc:260] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:33.540706 22848 ts_tablet_manager.cc:1398] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240502 01:53:33.541543 22036 tablet.cc:2369] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:53:33.576728 21904 tablet.cc:2369] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:33.593482 22854 raft_consensus.cc:482] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:33.593775 22854 raft_consensus.cc:504] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:33.595697 22854 leader_election.cc:288] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:33.604053 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cddc036193de43c9b7569f664f3e4d7c" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:33.604202 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cddc036193de43c9b7569f664f3e4d7c" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:33.604677 22120 raft_consensus.cc:2454] T cddc036193de43c9b7569f664f3e4d7c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:33.604735 22251 raft_consensus.cc:2454] T cddc036193de43c9b7569f664f3e4d7c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:33.605381 21792 leader_election.cc:302] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:33.605851 22854 raft_consensus.cc:2781] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:33.606010 22854 raft_consensus.cc:482] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:33.606122 22854 raft_consensus.cc:3037] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.608335 22854 raft_consensus.cc:504] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:33.608997 22854 leader_election.cc:288] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:33.609508 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cddc036193de43c9b7569f664f3e4d7c" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:33.609551 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cddc036193de43c9b7569f664f3e4d7c" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:33.609719 22120 raft_consensus.cc:3037] T cddc036193de43c9b7569f664f3e4d7c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.609755 22251 raft_consensus.cc:3037] T cddc036193de43c9b7569f664f3e4d7c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.611917 22120 raft_consensus.cc:2454] T cddc036193de43c9b7569f664f3e4d7c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:33.612066 22251 raft_consensus.cc:2454] T cddc036193de43c9b7569f664f3e4d7c P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:33.612399 21792 leader_election.cc:302] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:33.612735 22854 raft_consensus.cc:2781] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:33.613173 22854 raft_consensus.cc:686] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:53:33.613732 22854 consensus_queue.cc:237] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:33.618564 21715 catalog_manager.cc:5484] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:53:33.657529 22856 raft_consensus.cc:482] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:33.657799 22856 raft_consensus.cc:504] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:33.659476 22856 leader_election.cc:288] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:33.666059 22857 raft_consensus.cc:482] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:33.666307 22857 raft_consensus.cc:504] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:33.667245 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fa4e82b6a44b5bbc0f689a318090f7" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:33.667820 21989 raft_consensus.cc:2454] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:33.667683 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fa4e82b6a44b5bbc0f689a318090f7" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:33.668081 21857 raft_consensus.cc:2454] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:33.668495 22857 leader_election.cc:288] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:33.668987 22053 leader_election.cc:302] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:33.669380 22856 raft_consensus.cc:2781] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:33.669515 22856 raft_consensus.cc:482] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:33.669628 22856 raft_consensus.cc:3037] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.671679 22856 raft_consensus.cc:504] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:33.672313 22856 leader_election.cc:288] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:33.672940 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fa4e82b6a44b5bbc0f689a318090f7" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:33.672926 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fa4e82b6a44b5bbc0f689a318090f7" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:33.673156 21857 raft_consensus.cc:3037] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.673161 21989 raft_consensus.cc:3037] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.676332 21989 raft_consensus.cc:2454] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:33.676334 21857 raft_consensus.cc:2454] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:33.676766 22053 leader_election.cc:302] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:33.677131 22856 raft_consensus.cc:2781] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:33.678071 22856 raft_consensus.cc:686] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:53:33.678767 22856 consensus_queue.cc:237] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:33.684701 21715 catalog_manager.cc:5484] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:53:33.687304 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b02ca6aec3774fe5a1aff6ee307cbad5" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:33.687611 21857 raft_consensus.cc:2454] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:33.688105 22184 leader_election.cc:302] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:33.688489 22857 raft_consensus.cc:2781] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:33.688622 22857 raft_consensus.cc:482] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:33.688715 22857 raft_consensus.cc:3037] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.688907 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b02ca6aec3774fe5a1aff6ee307cbad5" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:33.689181 21989 raft_consensus.cc:2454] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:33.691139 22857 raft_consensus.cc:504] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:33.691740 22857 leader_election.cc:288] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:33.692243 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b02ca6aec3774fe5a1aff6ee307cbad5" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:33.692262 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b02ca6aec3774fe5a1aff6ee307cbad5" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:33.692487 21857 raft_consensus.cc:3037] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.692485 21989 raft_consensus.cc:3037] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.695410 21989 raft_consensus.cc:2454] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:33.695416 21857 raft_consensus.cc:2454] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:33.695837 22184 leader_election.cc:302] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:33.696166 22857 raft_consensus.cc:2781] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:33.696624 22857 raft_consensus.cc:686] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:53:33.697147 22857 consensus_queue.cc:237] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:33.701726 21715 catalog_manager.cc:5484] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
W20240502 01:53:33.742332 22298 tablet.cc:2369] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:53:34.084009 22875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:34.084398 22875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:34.092120 22875 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:53:34.105654 22854 consensus_queue.cc:1035] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:34.105720 22856 consensus_queue.cc:1035] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:34.116814 22870 consensus_queue.cc:1035] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:34.117624 22865 consensus_queue.cc:1035] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240502 01:53:34.260318 21989 raft_consensus.cc:1261] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:34.261394 22857 consensus_queue.cc:1035] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:34.270215 21857 raft_consensus.cc:1261] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:34.281597 22857 consensus_queue.cc:1035] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:34.367884 22929 mvcc.cc:205] Tried to move back new op lower bound from 7023062279191969792 to 7023062276907376640. Current Snapshot: MvccSnapshot[applied={T|T < 7023062279191969792}]
INSERT report
rows total: 2000
time total: 297.283 ms
time per row: 0.148641 ms
I20240502 01:53:34.560288 21706 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:34.570772 22100 tablet_service.cc:1467] Processing CreateTablet for tablet dcc8171c991b48ddbfda8007e3a87a38 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eaf29e0201fc4b738b1b30c13a773ffd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:34.570807 22231 tablet_service.cc:1467] Processing CreateTablet for tablet dcc8171c991b48ddbfda8007e3a87a38 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eaf29e0201fc4b738b1b30c13a773ffd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:34.571282 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc8171c991b48ddbfda8007e3a87a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:34.571282 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc8171c991b48ddbfda8007e3a87a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:34.571259 21969 tablet_service.cc:1467] Processing CreateTablet for tablet dcc8171c991b48ddbfda8007e3a87a38 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eaf29e0201fc4b738b1b30c13a773ffd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:34.571627 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc8171c991b48ddbfda8007e3a87a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:34.574396 21968 tablet_service.cc:1467] Processing CreateTablet for tablet 0eed9ee06a494588a32b4c951d6d3a14 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eaf29e0201fc4b738b1b30c13a773ffd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:34.574417 21835 tablet_service.cc:1467] Processing CreateTablet for tablet 3cf315f0117444ae9c5390a1e19460d9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eaf29e0201fc4b738b1b30c13a773ffd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:34.574851 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eed9ee06a494588a32b4c951d6d3a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:34.574903 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf315f0117444ae9c5390a1e19460d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:34.577916 22099 tablet_service.cc:1467] Processing CreateTablet for tablet 0eed9ee06a494588a32b4c951d6d3a14 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eaf29e0201fc4b738b1b30c13a773ffd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:34.578346 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eed9ee06a494588a32b4c951d6d3a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:34.578327 22230 tablet_service.cc:1467] Processing CreateTablet for tablet 0eed9ee06a494588a32b4c951d6d3a14 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eaf29e0201fc4b738b1b30c13a773ffd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:34.578899 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eed9ee06a494588a32b4c951d6d3a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:34.586367 22946 tablet_bootstrap.cc:492] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:34.587240 22098 tablet_service.cc:1467] Processing CreateTablet for tablet 3cf315f0117444ae9c5390a1e19460d9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eaf29e0201fc4b738b1b30c13a773ffd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:34.587880 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf315f0117444ae9c5390a1e19460d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:34.589380 22946 tablet_bootstrap.cc:654] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.595477 22947 tablet_bootstrap.cc:492] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:34.596323 22945 tablet_bootstrap.cc:492] T dcc8171c991b48ddbfda8007e3a87a38 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:34.598381 22945 tablet_bootstrap.cc:654] T dcc8171c991b48ddbfda8007e3a87a38 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.598381 22947 tablet_bootstrap.cc:654] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.604089 21969 tablet_service.cc:1467] Processing CreateTablet for tablet 3cf315f0117444ae9c5390a1e19460d9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eaf29e0201fc4b738b1b30c13a773ffd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:34.604756 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf315f0117444ae9c5390a1e19460d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:34.607532 22947 tablet_bootstrap.cc:492] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:34.607726 22947 ts_tablet_manager.cc:1367] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20240502 01:53:34.607868 22945 tablet_bootstrap.cc:492] T dcc8171c991b48ddbfda8007e3a87a38 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:34.608134 22945 ts_tablet_manager.cc:1367] T dcc8171c991b48ddbfda8007e3a87a38 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240502 01:53:34.608673 22947 raft_consensus.cc:348] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:34.608930 22947 raft_consensus.cc:374] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.609037 22947 raft_consensus.cc:729] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.609320 22945 raft_consensus.cc:348] T dcc8171c991b48ddbfda8007e3a87a38 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:34.609354 22947 consensus_queue.cc:260] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:34.609658 22945 raft_consensus.cc:374] T dcc8171c991b48ddbfda8007e3a87a38 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.609804 22945 raft_consensus.cc:729] T dcc8171c991b48ddbfda8007e3a87a38 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.609994 22947 ts_tablet_manager.cc:1398] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:34.610215 22945 consensus_queue.cc:260] T dcc8171c991b48ddbfda8007e3a87a38 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:34.611018 22945 ts_tablet_manager.cc:1398] T dcc8171c991b48ddbfda8007e3a87a38 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:53:34.611375 22945 tablet_bootstrap.cc:492] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:34.613137 22945 tablet_bootstrap.cc:654] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.623461 22947 tablet_bootstrap.cc:492] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:34.626026 22947 tablet_bootstrap.cc:654] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.628417 22951 tablet_bootstrap.cc:492] T 3cf315f0117444ae9c5390a1e19460d9 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:34.631182 22951 tablet_bootstrap.cc:654] T 3cf315f0117444ae9c5390a1e19460d9 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.633849 22946 tablet_bootstrap.cc:492] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:34.634030 22946 ts_tablet_manager.cc:1367] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.048s user 0.020s sys 0.015s
I20240502 01:53:34.634281 22945 tablet_bootstrap.cc:492] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:34.634568 22945 ts_tablet_manager.cc:1367] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.003s
I20240502 01:53:34.635157 22951 tablet_bootstrap.cc:492] T 3cf315f0117444ae9c5390a1e19460d9 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:34.635057 22946 raft_consensus.cc:348] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:34.635306 22946 raft_consensus.cc:374] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.635406 22946 raft_consensus.cc:729] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.635584 22951 ts_tablet_manager.cc:1367] T 3cf315f0117444ae9c5390a1e19460d9 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 01:53:34.635720 22945 raft_consensus.cc:348] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:34.635742 22946 consensus_queue.cc:260] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:34.635965 22945 raft_consensus.cc:374] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.636047 22945 raft_consensus.cc:729] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.636531 22946 ts_tablet_manager.cc:1398] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:53:34.636912 22946 tablet_bootstrap.cc:492] T dcc8171c991b48ddbfda8007e3a87a38 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:34.636823 22951 raft_consensus.cc:348] T 3cf315f0117444ae9c5390a1e19460d9 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:34.637195 22951 raft_consensus.cc:374] T 3cf315f0117444ae9c5390a1e19460d9 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.637137 22945 consensus_queue.cc:260] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:34.637416 22951 raft_consensus.cc:729] T 3cf315f0117444ae9c5390a1e19460d9 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.637857 22945 ts_tablet_manager.cc:1398] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 01:53:34.637771 22951 consensus_queue.cc:260] T 3cf315f0117444ae9c5390a1e19460d9 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:34.638211 22945 tablet_bootstrap.cc:492] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:34.638682 22946 tablet_bootstrap.cc:654] T dcc8171c991b48ddbfda8007e3a87a38 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.639720 22951 ts_tablet_manager.cc:1398] T 3cf315f0117444ae9c5390a1e19460d9 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:34.639822 22945 tablet_bootstrap.cc:654] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.646235 22947 tablet_bootstrap.cc:492] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:34.646620 22947 ts_tablet_manager.cc:1367] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.004s
I20240502 01:53:34.647826 22947 raft_consensus.cc:348] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:34.648277 22947 raft_consensus.cc:374] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.648427 22947 raft_consensus.cc:729] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.648826 22945 tablet_bootstrap.cc:492] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:34.649017 22945 ts_tablet_manager.cc:1367] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240502 01:53:34.649267 22947 consensus_queue.cc:260] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:34.649880 22947 ts_tablet_manager.cc:1398] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:34.649817 22945 raft_consensus.cc:348] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:34.650018 22945 raft_consensus.cc:374] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.650110 22945 raft_consensus.cc:729] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.650120 22946 tablet_bootstrap.cc:492] T dcc8171c991b48ddbfda8007e3a87a38 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:34.650285 22946 ts_tablet_manager.cc:1367] T dcc8171c991b48ddbfda8007e3a87a38 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.004s
I20240502 01:53:34.650377 22945 consensus_queue.cc:260] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:34.651077 22945 ts_tablet_manager.cc:1398] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:34.651095 22946 raft_consensus.cc:348] T dcc8171c991b48ddbfda8007e3a87a38 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:34.651311 22946 raft_consensus.cc:374] T dcc8171c991b48ddbfda8007e3a87a38 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.651401 22946 raft_consensus.cc:729] T dcc8171c991b48ddbfda8007e3a87a38 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.651657 22946 consensus_queue.cc:260] T dcc8171c991b48ddbfda8007e3a87a38 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:34.652164 22946 ts_tablet_manager.cc:1398] T dcc8171c991b48ddbfda8007e3a87a38 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:34.652474 22946 tablet_bootstrap.cc:492] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:34.653918 22946 tablet_bootstrap.cc:654] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.656008 22946 tablet_bootstrap.cc:492] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:34.656234 22946 ts_tablet_manager.cc:1367] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:34.656867 22946 raft_consensus.cc:348] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:34.657140 22946 raft_consensus.cc:374] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.657266 22946 raft_consensus.cc:729] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.657480 22946 consensus_queue.cc:260] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:34.658030 22946 ts_tablet_manager.cc:1398] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:34.676930 22870 raft_consensus.cc:482] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:34.677196 22870 raft_consensus.cc:504] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:34.677942 22870 leader_election.cc:288] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:34.678683 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf315f0117444ae9c5390a1e19460d9" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:34.678699 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf315f0117444ae9c5390a1e19460d9" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:34.678917 21856 raft_consensus.cc:2454] T 3cf315f0117444ae9c5390a1e19460d9 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:34.678946 21989 raft_consensus.cc:2454] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:34.679450 22053 leader_election.cc:302] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:34.679749 22870 raft_consensus.cc:2781] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:34.679857 22870 raft_consensus.cc:482] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:34.679970 22870 raft_consensus.cc:3037] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.681972 22870 raft_consensus.cc:504] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:34.682565 22870 leader_election.cc:288] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:34.683105 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf315f0117444ae9c5390a1e19460d9" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:34.683130 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf315f0117444ae9c5390a1e19460d9" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:34.683310 21856 raft_consensus.cc:3037] T 3cf315f0117444ae9c5390a1e19460d9 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.683353 21989 raft_consensus.cc:3037] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.685330 21989 raft_consensus.cc:2454] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:34.685345 21856 raft_consensus.cc:2454] T 3cf315f0117444ae9c5390a1e19460d9 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:34.685748 22053 leader_election.cc:302] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:34.686035 22870 raft_consensus.cc:2781] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:34.686228 22870 raft_consensus.cc:686] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:53:34.686499 22870 consensus_queue.cc:237] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:34.689409 21709 catalog_manager.cc:5484] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:53:34.745839 22870 raft_consensus.cc:482] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:34.746043 22870 raft_consensus.cc:504] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:34.746863 22870 leader_election.cc:288] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:34.747548 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eed9ee06a494588a32b4c951d6d3a14" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:34.747840 21989 raft_consensus.cc:2454] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:34.748301 22053 leader_election.cc:302] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:34.748663 22870 raft_consensus.cc:2781] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:34.748777 22870 raft_consensus.cc:482] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:34.748922 22870 raft_consensus.cc:3037] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.751674 22870 raft_consensus.cc:504] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:34.752362 22870 leader_election.cc:288] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:34.752964 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eed9ee06a494588a32b4c951d6d3a14" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:34.753170 21989 raft_consensus.cc:3037] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.754964 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eed9ee06a494588a32b4c951d6d3a14" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:34.755246 22251 raft_consensus.cc:2454] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:34.755096 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eed9ee06a494588a32b4c951d6d3a14" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:34.755412 22250 raft_consensus.cc:3037] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.755769 21989 raft_consensus.cc:2454] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:34.756194 22053 leader_election.cc:302] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:34.756505 22870 raft_consensus.cc:2781] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:34.756672 22870 raft_consensus.cc:686] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:53:34.757023 22870 consensus_queue.cc:237] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:34.757751 22250 raft_consensus.cc:2454] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:34.759899 21709 catalog_manager.cc:5484] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:53:34.808507 22857 raft_consensus.cc:482] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:34.808789 22857 raft_consensus.cc:504] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:34.809593 22857 leader_election.cc:288] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:34.810326 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc8171c991b48ddbfda8007e3a87a38" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:34.810639 21989 raft_consensus.cc:2454] T dcc8171c991b48ddbfda8007e3a87a38 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:34.811100 22184 leader_election.cc:302] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:34.811393 22857 raft_consensus.cc:2781] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:34.811501 22857 raft_consensus.cc:482] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:34.811648 22857 raft_consensus.cc:3037] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.813764 22857 raft_consensus.cc:504] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:34.814476 22857 leader_election.cc:288] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:34.815142 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc8171c991b48ddbfda8007e3a87a38" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:34.815424 21989 raft_consensus.cc:3037] T dcc8171c991b48ddbfda8007e3a87a38 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.817070 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc8171c991b48ddbfda8007e3a87a38" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:34.817216 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc8171c991b48ddbfda8007e3a87a38" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:34.817406 22120 raft_consensus.cc:2454] T dcc8171c991b48ddbfda8007e3a87a38 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:34.819918 21989 raft_consensus.cc:2454] T dcc8171c991b48ddbfda8007e3a87a38 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:34.820367 22184 leader_election.cc:302] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters: c64840657cdf4b93980bd34120c94d9a
I20240502 01:53:34.820719 22857 raft_consensus.cc:2781] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:34.820947 22857 raft_consensus.cc:686] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:53:34.821254 22857 consensus_queue.cc:237] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:34.824250 21709 catalog_manager.cc:5484] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:53:35.136497 22870 consensus_queue.cc:1035] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:35.153203 22968 consensus_queue.cc:1035] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:35.212668 22964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:35.213109 22964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:35.220969 22964 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:53:35.243867 22119 raft_consensus.cc:3037] T dcc8171c991b48ddbfda8007e3a87a38 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.247818 22857 consensus_queue.cc:1035] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240502 01:53:35.268196 22925 consensus_queue.cc:1035] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:35.292346 22968 consensus_queue.cc:1035] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:35.298381 22968 consensus_queue.cc:1035] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 266.698 ms
time per row: 0.133349 ms
I20240502 01:53:35.657096 21706 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:35.666834 21835 tablet_service.cc:1467] Processing CreateTablet for tablet ae6c263bd6ab4cc48e589c8a18ef19a4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c9e840e3d5c343ff9dac7e30dd1febee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.666993 22230 tablet_service.cc:1467] Processing CreateTablet for tablet ae6c263bd6ab4cc48e589c8a18ef19a4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c9e840e3d5c343ff9dac7e30dd1febee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.667200 21969 tablet_service.cc:1467] Processing CreateTablet for tablet ae6c263bd6ab4cc48e589c8a18ef19a4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c9e840e3d5c343ff9dac7e30dd1febee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.667387 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae6c263bd6ab4cc48e589c8a18ef19a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.667387 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae6c263bd6ab4cc48e589c8a18ef19a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.667670 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae6c263bd6ab4cc48e589c8a18ef19a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.672721 23027 tablet_bootstrap.cc:492] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:35.673396 22098 tablet_service.cc:1467] Processing CreateTablet for tablet 395c97e04948487cb64b0f3a4bf6f53c (DEFAULT_TABLE table=rebalance_test_table_2 [id=c9e840e3d5c343ff9dac7e30dd1febee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.673910 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 395c97e04948487cb64b0f3a4bf6f53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.674669 23027 tablet_bootstrap.cc:654] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.677873 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 395c97e04948487cb64b0f3a4bf6f53c (DEFAULT_TABLE table=rebalance_test_table_2 [id=c9e840e3d5c343ff9dac7e30dd1febee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.678138 22229 tablet_service.cc:1467] Processing CreateTablet for tablet a65fc966b76c4abe9fda77fe8938e4b9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c9e840e3d5c343ff9dac7e30dd1febee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.678411 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 395c97e04948487cb64b0f3a4bf6f53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.678792 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a65fc966b76c4abe9fda77fe8938e4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.678874 21837 tablet_service.cc:1467] Processing CreateTablet for tablet 395c97e04948487cb64b0f3a4bf6f53c (DEFAULT_TABLE table=rebalance_test_table_2 [id=c9e840e3d5c343ff9dac7e30dd1febee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.679100 21968 tablet_service.cc:1467] Processing CreateTablet for tablet a65fc966b76c4abe9fda77fe8938e4b9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c9e840e3d5c343ff9dac7e30dd1febee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.679282 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 395c97e04948487cb64b0f3a4bf6f53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.679504 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a65fc966b76c4abe9fda77fe8938e4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.681252 21836 tablet_service.cc:1467] Processing CreateTablet for tablet a65fc966b76c4abe9fda77fe8938e4b9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c9e840e3d5c343ff9dac7e30dd1febee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.681596 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a65fc966b76c4abe9fda77fe8938e4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.690798 23031 tablet_bootstrap.cc:492] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:35.692740 23031 tablet_bootstrap.cc:654] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.688311 23028 tablet_bootstrap.cc:492] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:35.694095 23029 tablet_bootstrap.cc:492] T ae6c263bd6ab4cc48e589c8a18ef19a4 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:35.694927 23028 tablet_bootstrap.cc:654] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.694926 23027 tablet_bootstrap.cc:492] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:35.695233 23027 ts_tablet_manager.cc:1367] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.006s
I20240502 01:53:35.696040 23029 tablet_bootstrap.cc:654] T ae6c263bd6ab4cc48e589c8a18ef19a4 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.696522 23027 raft_consensus.cc:348] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:35.698578 23027 raft_consensus.cc:374] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.698783 23027 raft_consensus.cc:729] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.699236 23027 consensus_queue.cc:260] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:35.700192 23028 tablet_bootstrap.cc:492] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:35.700356 23028 ts_tablet_manager.cc:1367] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20240502 01:53:35.700558 23027 ts_tablet_manager.cc:1398] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240502 01:53:35.701392 23028 raft_consensus.cc:348] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:35.701668 23028 raft_consensus.cc:374] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.701769 23028 raft_consensus.cc:729] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.702142 23028 consensus_queue.cc:260] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:35.703035 23028 ts_tablet_manager.cc:1398] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 01:53:35.703430 23028 tablet_bootstrap.cc:492] T 395c97e04948487cb64b0f3a4bf6f53c P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:35.705526 23028 tablet_bootstrap.cc:654] T 395c97e04948487cb64b0f3a4bf6f53c P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.706714 23027 tablet_bootstrap.cc:492] T a65fc966b76c4abe9fda77fe8938e4b9 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:35.707291 22865 raft_consensus.cc:482] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:35.707490 22865 raft_consensus.cc:504] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:35.707811 23029 tablet_bootstrap.cc:492] T ae6c263bd6ab4cc48e589c8a18ef19a4 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:35.707965 23029 ts_tablet_manager.cc:1367] T ae6c263bd6ab4cc48e589c8a18ef19a4 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20240502 01:53:35.708379 22865 leader_election.cc:288] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:35.708557 23027 tablet_bootstrap.cc:654] T a65fc966b76c4abe9fda77fe8938e4b9 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.708853 23029 raft_consensus.cc:348] T ae6c263bd6ab4cc48e589c8a18ef19a4 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:35.709123 23029 raft_consensus.cc:374] T ae6c263bd6ab4cc48e589c8a18ef19a4 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.709071 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6c263bd6ab4cc48e589c8a18ef19a4" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:35.709221 23029 raft_consensus.cc:729] T ae6c263bd6ab4cc48e589c8a18ef19a4 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.709465 23029 consensus_queue.cc:260] T ae6c263bd6ab4cc48e589c8a18ef19a4 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:35.709682 23031 tablet_bootstrap.cc:492] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:35.709779 22250 raft_consensus.cc:2454] T ae6c263bd6ab4cc48e589c8a18ef19a4 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:35.709857 23031 ts_tablet_manager.cc:1367] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.008s
I20240502 01:53:35.710279 21792 leader_election.cc:302] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:35.710598 22865 raft_consensus.cc:2781] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:35.710706 22865 raft_consensus.cc:482] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:35.710819 22865 raft_consensus.cc:3037] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.710758 23031 raft_consensus.cc:348] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:35.710970 23031 raft_consensus.cc:374] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.711055 23031 raft_consensus.cc:729] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.711331 23031 consensus_queue.cc:260] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:35.711894 23031 ts_tablet_manager.cc:1398] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:35.714921 22865 raft_consensus.cc:504] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:35.716446 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6c263bd6ab4cc48e589c8a18ef19a4" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:35.716774 22250 raft_consensus.cc:3037] T ae6c263bd6ab4cc48e589c8a18ef19a4 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.720717 22250 raft_consensus.cc:2454] T ae6c263bd6ab4cc48e589c8a18ef19a4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:35.721405 22865 leader_election.cc:288] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:35.722103 21792 leader_election.cc:302] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:35.722571 22865 raft_consensus.cc:2781] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:35.724431 23029 ts_tablet_manager.cc:1398] T ae6c263bd6ab4cc48e589c8a18ef19a4 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240502 01:53:35.724848 23029 tablet_bootstrap.cc:492] T a65fc966b76c4abe9fda77fe8938e4b9 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:35.728683 23029 tablet_bootstrap.cc:654] T a65fc966b76c4abe9fda77fe8938e4b9 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.729919 22865 raft_consensus.cc:686] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:53:35.730379 22865 consensus_queue.cc:237] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:35.743639 23029 tablet_bootstrap.cc:492] T a65fc966b76c4abe9fda77fe8938e4b9 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:35.743907 23029 ts_tablet_manager.cc:1367] T a65fc966b76c4abe9fda77fe8938e4b9 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240502 01:53:35.743598 21707 catalog_manager.cc:5484] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:53:35.745086 23029 raft_consensus.cc:348] T a65fc966b76c4abe9fda77fe8938e4b9 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:35.745421 23029 raft_consensus.cc:374] T a65fc966b76c4abe9fda77fe8938e4b9 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.745522 23029 raft_consensus.cc:729] T a65fc966b76c4abe9fda77fe8938e4b9 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.745903 23029 consensus_queue.cc:260] T a65fc966b76c4abe9fda77fe8938e4b9 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:35.748217 23029 ts_tablet_manager.cc:1398] T a65fc966b76c4abe9fda77fe8938e4b9 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:35.748606 23029 tablet_bootstrap.cc:492] T 395c97e04948487cb64b0f3a4bf6f53c P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:35.752918 23028 tablet_bootstrap.cc:492] T 395c97e04948487cb64b0f3a4bf6f53c P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:35.753077 23027 tablet_bootstrap.cc:492] T a65fc966b76c4abe9fda77fe8938e4b9 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:35.753134 23028 ts_tablet_manager.cc:1367] T 395c97e04948487cb64b0f3a4bf6f53c P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.050s user 0.016s sys 0.026s
I20240502 01:53:35.753270 23027 ts_tablet_manager.cc:1367] T a65fc966b76c4abe9fda77fe8938e4b9 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.035s
I20240502 01:53:35.754019 23028 raft_consensus.cc:348] T 395c97e04948487cb64b0f3a4bf6f53c P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:35.754173 23027 raft_consensus.cc:348] T a65fc966b76c4abe9fda77fe8938e4b9 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:35.754310 23028 raft_consensus.cc:374] T 395c97e04948487cb64b0f3a4bf6f53c P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.754408 23027 raft_consensus.cc:374] T a65fc966b76c4abe9fda77fe8938e4b9 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.754400 23028 raft_consensus.cc:729] T 395c97e04948487cb64b0f3a4bf6f53c P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.754494 23027 raft_consensus.cc:729] T a65fc966b76c4abe9fda77fe8938e4b9 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.754745 23028 consensus_queue.cc:260] T 395c97e04948487cb64b0f3a4bf6f53c P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:35.754792 23027 consensus_queue.cc:260] T a65fc966b76c4abe9fda77fe8938e4b9 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:35.755439 23028 ts_tablet_manager.cc:1398] T 395c97e04948487cb64b0f3a4bf6f53c P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:35.755512 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6c263bd6ab4cc48e589c8a18ef19a4" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:35.755759 23028 tablet_bootstrap.cc:492] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:35.755734 21989 raft_consensus.cc:2454] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:35.756106 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6c263bd6ab4cc48e589c8a18ef19a4" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:35.756299 21988 raft_consensus.cc:3037] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.756574 23029 tablet_bootstrap.cc:654] T 395c97e04948487cb64b0f3a4bf6f53c P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.757445 23028 tablet_bootstrap.cc:654] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.758116 23027 ts_tablet_manager.cc:1398] T a65fc966b76c4abe9fda77fe8938e4b9 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:53:35.759315 21988 raft_consensus.cc:2454] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:35.762492 23028 tablet_bootstrap.cc:492] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:35.762534 23029 tablet_bootstrap.cc:492] T 395c97e04948487cb64b0f3a4bf6f53c P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:35.762655 23028 ts_tablet_manager.cc:1367] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240502 01:53:35.762670 23029 ts_tablet_manager.cc:1367] T 395c97e04948487cb64b0f3a4bf6f53c P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.001s
I20240502 01:53:35.763454 23029 raft_consensus.cc:348] T 395c97e04948487cb64b0f3a4bf6f53c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:35.763454 23028 raft_consensus.cc:348] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:35.763670 23028 raft_consensus.cc:374] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.763665 23029 raft_consensus.cc:374] T 395c97e04948487cb64b0f3a4bf6f53c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.763754 23029 raft_consensus.cc:729] T 395c97e04948487cb64b0f3a4bf6f53c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.763764 23028 raft_consensus.cc:729] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.763999 23028 consensus_queue.cc:260] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:35.763988 23029 consensus_queue.cc:260] T 395c97e04948487cb64b0f3a4bf6f53c P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:35.764573 23028 ts_tablet_manager.cc:1398] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:35.764640 23029 ts_tablet_manager.cc:1398] T 395c97e04948487cb64b0f3a4bf6f53c P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:35.805187 22932 raft_consensus.cc:482] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:35.805415 22932 raft_consensus.cc:504] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:35.806154 22932 leader_election.cc:288] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:35.806746 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395c97e04948487cb64b0f3a4bf6f53c" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:35.806813 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395c97e04948487cb64b0f3a4bf6f53c" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:35.807053 21856 raft_consensus.cc:2454] T 395c97e04948487cb64b0f3a4bf6f53c P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:35.807076 22250 raft_consensus.cc:2454] T 395c97e04948487cb64b0f3a4bf6f53c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:35.807590 22053 leader_election.cc:302] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:35.807870 22932 raft_consensus.cc:2781] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:35.807994 22932 raft_consensus.cc:482] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:35.808130 22932 raft_consensus.cc:3037] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.810201 22932 raft_consensus.cc:504] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:35.810827 22932 leader_election.cc:288] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:35.811367 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395c97e04948487cb64b0f3a4bf6f53c" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:35.811421 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395c97e04948487cb64b0f3a4bf6f53c" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:35.811568 22250 raft_consensus.cc:3037] T 395c97e04948487cb64b0f3a4bf6f53c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.811682 21856 raft_consensus.cc:3037] T 395c97e04948487cb64b0f3a4bf6f53c P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.813620 22250 raft_consensus.cc:2454] T 395c97e04948487cb64b0f3a4bf6f53c P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:35.813653 21856 raft_consensus.cc:2454] T 395c97e04948487cb64b0f3a4bf6f53c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:35.814155 22055 leader_election.cc:302] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:35.814522 22932 raft_consensus.cc:2781] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:35.814680 22932 raft_consensus.cc:686] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:53:35.814955 22932 consensus_queue.cc:237] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:35.817929 21706 catalog_manager.cc:5484] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:53:35.885556 22865 raft_consensus.cc:482] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:35.885741 22865 raft_consensus.cc:504] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:35.886431 22865 leader_election.cc:288] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:35.886931 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a65fc966b76c4abe9fda77fe8938e4b9" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:35.886960 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a65fc966b76c4abe9fda77fe8938e4b9" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:35.887182 22250 raft_consensus.cc:2454] T a65fc966b76c4abe9fda77fe8938e4b9 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:35.887235 21988 raft_consensus.cc:2454] T a65fc966b76c4abe9fda77fe8938e4b9 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:35.887624 21792 leader_election.cc:302] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:35.887914 22865 raft_consensus.cc:2781] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:35.888016 22865 raft_consensus.cc:482] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:35.888084 22865 raft_consensus.cc:3037] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.890395 22865 raft_consensus.cc:504] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:35.891032 22865 leader_election.cc:288] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:35.891492 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a65fc966b76c4abe9fda77fe8938e4b9" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:35.891536 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a65fc966b76c4abe9fda77fe8938e4b9" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:35.891713 21988 raft_consensus.cc:3037] T a65fc966b76c4abe9fda77fe8938e4b9 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.891734 22250 raft_consensus.cc:3037] T a65fc966b76c4abe9fda77fe8938e4b9 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.893622 22250 raft_consensus.cc:2454] T a65fc966b76c4abe9fda77fe8938e4b9 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:35.893651 21988 raft_consensus.cc:2454] T a65fc966b76c4abe9fda77fe8938e4b9 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:35.894028 21792 leader_election.cc:302] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:35.894353 22865 raft_consensus.cc:2781] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:35.894483 22865 raft_consensus.cc:686] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:53:35.894788 22865 consensus_queue.cc:237] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:35.898573 21706 catalog_manager.cc:5484] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:53:36.237720 22932 consensus_queue.cc:1035] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:36.239266 22865 consensus_queue.cc:1035] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:36.267665 23040 consensus_queue.cc:1035] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:36.282217 22870 consensus_queue.cc:1035] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:36.296052 23040 consensus_queue.cc:1035] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:36.313517 22865 consensus_queue.cc:1035] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:36.324709 23047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:36.325150 23047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:36.333765 23047 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 214.334 ms
time per row: 0.107167 ms
I20240502 01:53:36.701432 21707 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:36.710537 22098 tablet_service.cc:1467] Processing CreateTablet for tablet db96ccbe81564bb589b0de65977b3321 (DEFAULT_TABLE table=rebalance_test_table_3 [id=27064ffcfcc643d6ad082b6cd6ea21db]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.710741 21835 tablet_service.cc:1467] Processing CreateTablet for tablet db96ccbe81564bb589b0de65977b3321 (DEFAULT_TABLE table=rebalance_test_table_3 [id=27064ffcfcc643d6ad082b6cd6ea21db]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.711068 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db96ccbe81564bb589b0de65977b3321. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.711236 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db96ccbe81564bb589b0de65977b3321. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.711215 21968 tablet_service.cc:1467] Processing CreateTablet for tablet db96ccbe81564bb589b0de65977b3321 (DEFAULT_TABLE table=rebalance_test_table_3 [id=27064ffcfcc643d6ad082b6cd6ea21db]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.711625 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db96ccbe81564bb589b0de65977b3321. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.713639 22099 tablet_service.cc:1467] Processing CreateTablet for tablet da66e78c2c5044b2924682be0b643b0a (DEFAULT_TABLE table=rebalance_test_table_3 [id=27064ffcfcc643d6ad082b6cd6ea21db]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.714059 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da66e78c2c5044b2924682be0b643b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.716066 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 88457f21ad4a4e00bb372479c13963ec (DEFAULT_TABLE table=rebalance_test_table_3 [id=27064ffcfcc643d6ad082b6cd6ea21db]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.716148 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 88457f21ad4a4e00bb372479c13963ec (DEFAULT_TABLE table=rebalance_test_table_3 [id=27064ffcfcc643d6ad082b6cd6ea21db]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.716454 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88457f21ad4a4e00bb372479c13963ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.716607 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88457f21ad4a4e00bb372479c13963ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.721414 22230 tablet_service.cc:1467] Processing CreateTablet for tablet da66e78c2c5044b2924682be0b643b0a (DEFAULT_TABLE table=rebalance_test_table_3 [id=27064ffcfcc643d6ad082b6cd6ea21db]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.721937 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da66e78c2c5044b2924682be0b643b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.725795 21837 tablet_service.cc:1467] Processing CreateTablet for tablet da66e78c2c5044b2924682be0b643b0a (DEFAULT_TABLE table=rebalance_test_table_3 [id=27064ffcfcc643d6ad082b6cd6ea21db]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.726037 21969 tablet_service.cc:1467] Processing CreateTablet for tablet 88457f21ad4a4e00bb372479c13963ec (DEFAULT_TABLE table=rebalance_test_table_3 [id=27064ffcfcc643d6ad082b6cd6ea21db]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.726249 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da66e78c2c5044b2924682be0b643b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.726460 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88457f21ad4a4e00bb372479c13963ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.736606 23116 tablet_bootstrap.cc:492] T db96ccbe81564bb589b0de65977b3321 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:36.738111 23117 tablet_bootstrap.cc:492] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:36.738785 23115 tablet_bootstrap.cc:492] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:36.739457 23118 tablet_bootstrap.cc:492] T 88457f21ad4a4e00bb372479c13963ec P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:36.740008 23117 tablet_bootstrap.cc:654] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:36.740646 23116 tablet_bootstrap.cc:654] T db96ccbe81564bb589b0de65977b3321 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:36.740897 23115 tablet_bootstrap.cc:654] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:36.741518 23118 tablet_bootstrap.cc:654] T 88457f21ad4a4e00bb372479c13963ec P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:36.747010 23115 tablet_bootstrap.cc:492] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:36.747229 23115 ts_tablet_manager.cc:1367] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240502 01:53:36.748370 23115 raft_consensus.cc:348] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:36.748687 23115 raft_consensus.cc:374] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:36.748821 23115 raft_consensus.cc:729] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:36.749220 23115 consensus_queue.cc:260] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:36.751624 23115 ts_tablet_manager.cc:1398] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:36.752044 23115 tablet_bootstrap.cc:492] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:36.754393 23115 tablet_bootstrap.cc:654] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:36.757230 23118 tablet_bootstrap.cc:492] T 88457f21ad4a4e00bb372479c13963ec P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:36.757434 23118 ts_tablet_manager.cc:1367] T 88457f21ad4a4e00bb372479c13963ec P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.005s
I20240502 01:53:36.757848 23116 tablet_bootstrap.cc:492] T db96ccbe81564bb589b0de65977b3321 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:36.758034 23116 ts_tablet_manager.cc:1367] T db96ccbe81564bb589b0de65977b3321 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.007s
I20240502 01:53:36.758612 23118 raft_consensus.cc:348] T 88457f21ad4a4e00bb372479c13963ec P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:36.758941 23118 raft_consensus.cc:374] T 88457f21ad4a4e00bb372479c13963ec P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:36.758936 23116 raft_consensus.cc:348] T db96ccbe81564bb589b0de65977b3321 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:36.759065 23118 raft_consensus.cc:729] T 88457f21ad4a4e00bb372479c13963ec P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:36.759186 23116 raft_consensus.cc:374] T db96ccbe81564bb589b0de65977b3321 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:36.759305 23116 raft_consensus.cc:729] T db96ccbe81564bb589b0de65977b3321 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:36.759366 23118 consensus_queue.cc:260] T 88457f21ad4a4e00bb372479c13963ec P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:36.759584 23116 consensus_queue.cc:260] T db96ccbe81564bb589b0de65977b3321 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:36.760176 23118 ts_tablet_manager.cc:1398] T 88457f21ad4a4e00bb372479c13963ec P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240502 01:53:36.760329 23116 ts_tablet_manager.cc:1398] T db96ccbe81564bb589b0de65977b3321 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:36.760617 23116 tablet_bootstrap.cc:492] T da66e78c2c5044b2924682be0b643b0a P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:36.760550 23118 tablet_bootstrap.cc:492] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:36.762537 23116 tablet_bootstrap.cc:654] T da66e78c2c5044b2924682be0b643b0a P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:36.762800 23117 tablet_bootstrap.cc:492] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:36.762974 23118 tablet_bootstrap.cc:654] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:36.762974 23117 ts_tablet_manager.cc:1367] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.009s
I20240502 01:53:36.763970 23117 raft_consensus.cc:348] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:36.764289 23117 raft_consensus.cc:374] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:36.764395 23117 raft_consensus.cc:729] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:36.765808 23115 tablet_bootstrap.cc:492] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:36.766044 23115 ts_tablet_manager.cc:1367] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240502 01:53:36.767071 23115 raft_consensus.cc:348] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:36.767333 23115 raft_consensus.cc:374] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:36.767251 23117 consensus_queue.cc:260] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:36.767402 23115 raft_consensus.cc:729] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:36.767696 23115 consensus_queue.cc:260] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:36.768051 23117 ts_tablet_manager.cc:1398] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:53:36.768589 23118 tablet_bootstrap.cc:492] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:36.768635 23117 tablet_bootstrap.cc:492] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:36.768769 23118 ts_tablet_manager.cc:1367] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20240502 01:53:36.769701 23118 raft_consensus.cc:348] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:36.769922 23118 raft_consensus.cc:374] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:36.770004 23118 raft_consensus.cc:729] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:36.770259 23118 consensus_queue.cc:260] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:36.770784 23118 ts_tablet_manager.cc:1398] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:36.772192 23117 tablet_bootstrap.cc:654] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:36.775391 23116 tablet_bootstrap.cc:492] T da66e78c2c5044b2924682be0b643b0a P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:36.775758 23116 ts_tablet_manager.cc:1367] T da66e78c2c5044b2924682be0b643b0a P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20240502 01:53:36.776813 23116 raft_consensus.cc:348] T da66e78c2c5044b2924682be0b643b0a P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:36.777077 23116 raft_consensus.cc:374] T da66e78c2c5044b2924682be0b643b0a P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:36.777165 23116 raft_consensus.cc:729] T da66e78c2c5044b2924682be0b643b0a P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:36.777443 23116 consensus_queue.cc:260] T da66e78c2c5044b2924682be0b643b0a P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:36.778173 23116 ts_tablet_manager.cc:1398] T da66e78c2c5044b2924682be0b643b0a P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:36.778946 23115 ts_tablet_manager.cc:1398] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240502 01:53:36.784981 23115 tablet_bootstrap.cc:492] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:36.786811 23115 tablet_bootstrap.cc:654] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:36.789810 23115 tablet_bootstrap.cc:492] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:36.789968 23115 ts_tablet_manager.cc:1367] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 01:53:36.790201 23117 tablet_bootstrap.cc:492] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:36.790378 23117 ts_tablet_manager.cc:1367] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.011s
I20240502 01:53:36.790767 23115 raft_consensus.cc:348] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:36.790985 23115 raft_consensus.cc:374] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:36.791077 23115 raft_consensus.cc:729] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:36.791062 23117 raft_consensus.cc:348] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:36.791306 23117 raft_consensus.cc:374] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:36.791386 23117 raft_consensus.cc:729] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:36.791316 23115 consensus_queue.cc:260] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:36.791654 23117 consensus_queue.cc:260] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:36.791889 23115 ts_tablet_manager.cc:1398] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:36.792233 23117 ts_tablet_manager.cc:1398] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:36.806960 22892 raft_consensus.cc:482] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:36.807204 22892 raft_consensus.cc:504] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:36.808840 22892 leader_election.cc:288] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:36.815014 22925 raft_consensus.cc:482] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:36.815261 22925 raft_consensus.cc:504] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:36.816107 22925 leader_election.cc:288] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:36.816305 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db96ccbe81564bb589b0de65977b3321" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:36.816576 21856 raft_consensus.cc:2454] T db96ccbe81564bb589b0de65977b3321 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:36.816593 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da66e78c2c5044b2924682be0b643b0a" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:36.816816 21857 raft_consensus.cc:2454] T da66e78c2c5044b2924682be0b643b0a P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:36.816807 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db96ccbe81564bb589b0de65977b3321" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:36.817023 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da66e78c2c5044b2924682be0b643b0a" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:36.817090 22120 raft_consensus.cc:2454] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:36.817198 22119 raft_consensus.cc:2454] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:36.817209 21922 leader_election.cc:302] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:53:36.817615 22184 leader_election.cc:302] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:36.818023 22925 raft_consensus.cc:2781] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:36.818092 22892 raft_consensus.cc:2781] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:36.818136 22925 raft_consensus.cc:482] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:36.818241 22892 raft_consensus.cc:482] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:36.818281 22925 raft_consensus.cc:3037] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:36.818352 22892 raft_consensus.cc:3037] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:36.821337 22925 raft_consensus.cc:504] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:36.821364 22892 raft_consensus.cc:504] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:36.822110 22925 leader_election.cc:288] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:36.822474 22892 leader_election.cc:288] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:36.822574 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da66e78c2c5044b2924682be0b643b0a" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:36.822789 22119 raft_consensus.cc:3037] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:36.822732 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da66e78c2c5044b2924682be0b643b0a" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:36.822971 21857 raft_consensus.cc:3037] T da66e78c2c5044b2924682be0b643b0a P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:36.822997 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db96ccbe81564bb589b0de65977b3321" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:36.823220 22120 raft_consensus.cc:3037] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:36.825903 22119 raft_consensus.cc:2454] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:36.826128 21857 raft_consensus.cc:2454] T da66e78c2c5044b2924682be0b643b0a P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:36.826248 22120 raft_consensus.cc:2454] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:36.826391 22186 leader_election.cc:302] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:36.826545 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db96ccbe81564bb589b0de65977b3321" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:36.826797 21856 raft_consensus.cc:3037] T db96ccbe81564bb589b0de65977b3321 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:36.826821 22925 raft_consensus.cc:2781] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:36.827005 21924 leader_election.cc:302] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:36.827415 22925 raft_consensus.cc:686] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:53:36.827482 22892 raft_consensus.cc:2781] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:36.827828 22925 consensus_queue.cc:237] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:36.829414 21856 raft_consensus.cc:2454] T db96ccbe81564bb589b0de65977b3321 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:36.841984 22892 raft_consensus.cc:686] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:53:36.842597 22892 consensus_queue.cc:237] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:36.843488 21707 catalog_manager.cc:5484] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:53:36.846395 21715 catalog_manager.cc:5484] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:53:36.921808 22932 raft_consensus.cc:482] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:36.922080 22932 raft_consensus.cc:504] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:36.922823 22932 leader_election.cc:288] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:36.923295 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88457f21ad4a4e00bb372479c13963ec" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:36.923393 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88457f21ad4a4e00bb372479c13963ec" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:36.923552 21988 raft_consensus.cc:2454] T 88457f21ad4a4e00bb372479c13963ec P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:36.923643 22251 raft_consensus.cc:2454] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:36.924047 22053 leader_election.cc:302] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:36.924360 22932 raft_consensus.cc:2781] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:36.924463 22932 raft_consensus.cc:482] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:36.924530 22932 raft_consensus.cc:3037] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:36.926510 22932 raft_consensus.cc:504] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:36.927132 22932 leader_election.cc:288] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:36.927666 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88457f21ad4a4e00bb372479c13963ec" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:36.927680 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88457f21ad4a4e00bb372479c13963ec" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:36.927902 21988 raft_consensus.cc:3037] T 88457f21ad4a4e00bb372479c13963ec P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:36.927927 22251 raft_consensus.cc:3037] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:36.930830 22251 raft_consensus.cc:2454] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:36.930845 21988 raft_consensus.cc:2454] T 88457f21ad4a4e00bb372479c13963ec P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:36.931358 22055 leader_election.cc:302] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:36.931656 22932 raft_consensus.cc:2781] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:36.931815 22932 raft_consensus.cc:686] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:53:36.932134 22932 consensus_queue.cc:237] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:36.935956 21715 catalog_manager.cc:5484] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:53:37.256976 22892 consensus_queue.cc:1035] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:37.273288 23132 consensus_queue.cc:1035] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:37.324000 22925 consensus_queue.cc:1035] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:37.335539 22925 consensus_queue.cc:1035] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:37.343999 23136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:37.344372 23136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:37.352233 23136 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 01:53:37.428257 22932 consensus_queue.cc:1035] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:37.459599 22932 consensus_queue.cc:1035] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 253.922 ms
time per row: 0.126961 ms
I20240502 01:53:37.755411 21715 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:37.764184 21969 tablet_service.cc:1467] Processing CreateTablet for tablet 8e18fd328b0a45b98eadddb574737672 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0bafdd6983184b96aa892f802d62ebbc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:37.764459 21835 tablet_service.cc:1467] Processing CreateTablet for tablet 8e18fd328b0a45b98eadddb574737672 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0bafdd6983184b96aa892f802d62ebbc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:37.764731 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e18fd328b0a45b98eadddb574737672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:37.765295 22098 tablet_service.cc:1467] Processing CreateTablet for tablet 8e18fd328b0a45b98eadddb574737672 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0bafdd6983184b96aa892f802d62ebbc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:37.765347 22230 tablet_service.cc:1467] Processing CreateTablet for tablet 5ccc8d73d1ea4539a170022fe6ad890e (DEFAULT_TABLE table=rebalance_test_table_4 [id=0bafdd6983184b96aa892f802d62ebbc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:37.765693 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ccc8d73d1ea4539a170022fe6ad890e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:37.765730 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e18fd328b0a45b98eadddb574737672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:37.768047 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e18fd328b0a45b98eadddb574737672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:37.770279 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb056b49f4d44869749288abfd50d84 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0bafdd6983184b96aa892f802d62ebbc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:37.770792 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb056b49f4d44869749288abfd50d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:37.773339 23207 tablet_bootstrap.cc:492] T 8e18fd328b0a45b98eadddb574737672 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:37.773368 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb056b49f4d44869749288abfd50d84 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0bafdd6983184b96aa892f802d62ebbc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:37.773799 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb056b49f4d44869749288abfd50d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:37.779314 23209 tablet_bootstrap.cc:492] T 8e18fd328b0a45b98eadddb574737672 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:37.779529 21968 tablet_service.cc:1467] Processing CreateTablet for tablet 5ccc8d73d1ea4539a170022fe6ad890e (DEFAULT_TABLE table=rebalance_test_table_4 [id=0bafdd6983184b96aa892f802d62ebbc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:37.779968 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ccc8d73d1ea4539a170022fe6ad890e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:37.781083 23209 tablet_bootstrap.cc:654] T 8e18fd328b0a45b98eadddb574737672 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:37.781335 21837 tablet_service.cc:1467] Processing CreateTablet for tablet 5ccc8d73d1ea4539a170022fe6ad890e (DEFAULT_TABLE table=rebalance_test_table_4 [id=0bafdd6983184b96aa892f802d62ebbc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:37.781773 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ccc8d73d1ea4539a170022fe6ad890e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:37.785460 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb056b49f4d44869749288abfd50d84 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0bafdd6983184b96aa892f802d62ebbc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:37.785975 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb056b49f4d44869749288abfd50d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:37.794350 23207 tablet_bootstrap.cc:654] T 8e18fd328b0a45b98eadddb574737672 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:37.809307 23208 tablet_bootstrap.cc:492] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:37.811551 23208 tablet_bootstrap.cc:654] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:37.814162 23210 tablet_bootstrap.cc:492] T 5ccc8d73d1ea4539a170022fe6ad890e P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:37.815182 23208 tablet_bootstrap.cc:492] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:37.815413 23208 ts_tablet_manager.cc:1367] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240502 01:53:37.815462 23209 tablet_bootstrap.cc:492] T 8e18fd328b0a45b98eadddb574737672 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:37.815639 23209 ts_tablet_manager.cc:1367] T 8e18fd328b0a45b98eadddb574737672 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.037s user 0.017s sys 0.017s
I20240502 01:53:37.816475 23209 raft_consensus.cc:348] T 8e18fd328b0a45b98eadddb574737672 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:37.816700 23209 raft_consensus.cc:374] T 8e18fd328b0a45b98eadddb574737672 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:37.816776 23209 raft_consensus.cc:729] T 8e18fd328b0a45b98eadddb574737672 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:37.816864 23210 tablet_bootstrap.cc:654] T 5ccc8d73d1ea4539a170022fe6ad890e P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:37.817018 23209 consensus_queue.cc:260] T 8e18fd328b0a45b98eadddb574737672 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:37.818811 23208 raft_consensus.cc:348] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:37.820232 23207 tablet_bootstrap.cc:492] T 8e18fd328b0a45b98eadddb574737672 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:37.820401 23207 ts_tablet_manager.cc:1367] T 8e18fd328b0a45b98eadddb574737672 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.047s user 0.000s sys 0.010s
I20240502 01:53:37.821326 23208 raft_consensus.cc:374] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:37.821277 23207 raft_consensus.cc:348] T 8e18fd328b0a45b98eadddb574737672 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:37.821455 23208 raft_consensus.cc:729] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:37.821484 23207 raft_consensus.cc:374] T 8e18fd328b0a45b98eadddb574737672 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:37.821557 23207 raft_consensus.cc:729] T 8e18fd328b0a45b98eadddb574737672 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:37.821792 23207 consensus_queue.cc:260] T 8e18fd328b0a45b98eadddb574737672 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:37.821805 23208 consensus_queue.cc:260] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:37.822499 23208 ts_tablet_manager.cc:1398] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 01:53:37.826180 23207 ts_tablet_manager.cc:1398] T 8e18fd328b0a45b98eadddb574737672 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:53:37.826604 23207 tablet_bootstrap.cc:492] T 8eb056b49f4d44869749288abfd50d84 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:37.828433 23207 tablet_bootstrap.cc:654] T 8eb056b49f4d44869749288abfd50d84 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:37.830176 23209 ts_tablet_manager.cc:1398] T 8e18fd328b0a45b98eadddb574737672 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.014s user 0.000s sys 0.003s
I20240502 01:53:37.832007 23208 tablet_bootstrap.cc:492] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:37.833206 23209 tablet_bootstrap.cc:492] T 5ccc8d73d1ea4539a170022fe6ad890e P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:37.835009 23209 tablet_bootstrap.cc:654] T 5ccc8d73d1ea4539a170022fe6ad890e P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:37.838063 23208 tablet_bootstrap.cc:654] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:37.841079 23207 tablet_bootstrap.cc:492] T 8eb056b49f4d44869749288abfd50d84 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:37.841352 23207 ts_tablet_manager.cc:1367] T 8eb056b49f4d44869749288abfd50d84 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240502 01:53:37.842532 23207 raft_consensus.cc:348] T 8eb056b49f4d44869749288abfd50d84 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:37.842862 23207 raft_consensus.cc:374] T 8eb056b49f4d44869749288abfd50d84 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:37.842962 23207 raft_consensus.cc:729] T 8eb056b49f4d44869749288abfd50d84 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:37.843319 23207 consensus_queue.cc:260] T 8eb056b49f4d44869749288abfd50d84 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:37.844259 23207 ts_tablet_manager.cc:1398] T 8eb056b49f4d44869749288abfd50d84 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:53:37.844573 23215 raft_consensus.cc:482] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:37.844798 23215 raft_consensus.cc:504] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:37.846127 23215 leader_election.cc:288] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:37.847370 23210 tablet_bootstrap.cc:492] T 5ccc8d73d1ea4539a170022fe6ad890e P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:37.847257 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e18fd328b0a45b98eadddb574737672" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:37.847551 23210 ts_tablet_manager.cc:1367] T 5ccc8d73d1ea4539a170022fe6ad890e P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.011s
I20240502 01:53:37.847561 21988 raft_consensus.cc:2454] T 8e18fd328b0a45b98eadddb574737672 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:37.848212 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e18fd328b0a45b98eadddb574737672" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:37.848438 22120 raft_consensus.cc:2454] T 8e18fd328b0a45b98eadddb574737672 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:37.848640 23210 raft_consensus.cc:348] T 5ccc8d73d1ea4539a170022fe6ad890e P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:37.848914 23210 raft_consensus.cc:374] T 5ccc8d73d1ea4539a170022fe6ad890e P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:37.849061 23210 raft_consensus.cc:729] T 5ccc8d73d1ea4539a170022fe6ad890e P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:37.849437 21790 leader_election.cc:302] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:53:37.849359 23210 consensus_queue.cc:260] T 5ccc8d73d1ea4539a170022fe6ad890e P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:37.850158 23210 ts_tablet_manager.cc:1398] T 5ccc8d73d1ea4539a170022fe6ad890e P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:37.850494 23210 tablet_bootstrap.cc:492] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:37.852196 23210 tablet_bootstrap.cc:654] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:37.853521 23209 tablet_bootstrap.cc:492] T 5ccc8d73d1ea4539a170022fe6ad890e P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:37.853744 23209 ts_tablet_manager.cc:1367] T 5ccc8d73d1ea4539a170022fe6ad890e P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.006s
I20240502 01:53:37.854780 23209 raft_consensus.cc:348] T 5ccc8d73d1ea4539a170022fe6ad890e P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:37.855047 23209 raft_consensus.cc:374] T 5ccc8d73d1ea4539a170022fe6ad890e P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:37.855122 23209 raft_consensus.cc:729] T 5ccc8d73d1ea4539a170022fe6ad890e P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:37.855410 23210 tablet_bootstrap.cc:492] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:37.855412 23209 consensus_queue.cc:260] T 5ccc8d73d1ea4539a170022fe6ad890e P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:37.855561 23210 ts_tablet_manager.cc:1367] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 01:53:37.856312 23210 raft_consensus.cc:348] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:37.856555 23210 raft_consensus.cc:374] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:37.856581 23215 raft_consensus.cc:2781] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:37.856657 23210 raft_consensus.cc:729] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:37.856743 23215 raft_consensus.cc:482] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:37.856846 23209 ts_tablet_manager.cc:1398] T 5ccc8d73d1ea4539a170022fe6ad890e P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:37.856869 23215 raft_consensus.cc:3037] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.857053 23210 consensus_queue.cc:260] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:37.857604 23210 ts_tablet_manager.cc:1398] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:37.861382 23215 raft_consensus.cc:504] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:37.863404 23215 leader_election.cc:288] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:37.863337 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e18fd328b0a45b98eadddb574737672" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:37.863584 21988 raft_consensus.cc:3037] T 8e18fd328b0a45b98eadddb574737672 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.865422 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e18fd328b0a45b98eadddb574737672" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:37.865679 22120 raft_consensus.cc:3037] T 8e18fd328b0a45b98eadddb574737672 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.866919 21988 raft_consensus.cc:2454] T 8e18fd328b0a45b98eadddb574737672 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:37.866986 23208 tablet_bootstrap.cc:492] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:37.867179 23208 ts_tablet_manager.cc:1367] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.001s
I20240502 01:53:37.867424 21790 leader_election.cc:302] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:53:37.867935 23208 raft_consensus.cc:348] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:37.868125 23208 raft_consensus.cc:374] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:37.868213 23208 raft_consensus.cc:729] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:37.868410 22120 raft_consensus.cc:2454] T 8e18fd328b0a45b98eadddb574737672 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:37.868471 23208 consensus_queue.cc:260] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:37.869079 23208 ts_tablet_manager.cc:1398] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:37.869297 23215 raft_consensus.cc:2781] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:37.869504 23208 tablet_bootstrap.cc:492] T 8eb056b49f4d44869749288abfd50d84 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:37.869506 23215 raft_consensus.cc:686] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:53:37.869884 23215 consensus_queue.cc:237] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:37.871371 23208 tablet_bootstrap.cc:654] T 8eb056b49f4d44869749288abfd50d84 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:37.873522 23208 tablet_bootstrap.cc:492] T 8eb056b49f4d44869749288abfd50d84 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:37.873654 23208 ts_tablet_manager.cc:1367] T 8eb056b49f4d44869749288abfd50d84 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 01:53:37.873461 21711 catalog_manager.cc:5484] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:53:37.874290 23208 raft_consensus.cc:348] T 8eb056b49f4d44869749288abfd50d84 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:37.874478 23208 raft_consensus.cc:374] T 8eb056b49f4d44869749288abfd50d84 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:37.874596 23208 raft_consensus.cc:729] T 8eb056b49f4d44869749288abfd50d84 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:37.874828 23208 consensus_queue.cc:260] T 8eb056b49f4d44869749288abfd50d84 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:37.875383 23208 ts_tablet_manager.cc:1398] T 8eb056b49f4d44869749288abfd50d84 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:37.903131 23215 raft_consensus.cc:482] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:37.903393 23215 raft_consensus.cc:504] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:37.904124 23215 leader_election.cc:288] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:37.904778 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccc8d73d1ea4539a170022fe6ad890e" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:37.904847 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccc8d73d1ea4539a170022fe6ad890e" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:37.905071 22251 raft_consensus.cc:2454] T 5ccc8d73d1ea4539a170022fe6ad890e P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:37.905164 21988 raft_consensus.cc:2454] T 5ccc8d73d1ea4539a170022fe6ad890e P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:37.905586 21792 leader_election.cc:302] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:37.905939 23215 raft_consensus.cc:2781] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:37.906055 23215 raft_consensus.cc:482] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:37.906198 23215 raft_consensus.cc:3037] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.908262 23215 raft_consensus.cc:504] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:37.908829 23215 leader_election.cc:288] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:37.909281 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccc8d73d1ea4539a170022fe6ad890e" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:37.909359 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccc8d73d1ea4539a170022fe6ad890e" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:37.909490 22251 raft_consensus.cc:3037] T 5ccc8d73d1ea4539a170022fe6ad890e P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.909636 21988 raft_consensus.cc:3037] T 5ccc8d73d1ea4539a170022fe6ad890e P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.909822 23131 raft_consensus.cc:482] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:37.909972 23131 raft_consensus.cc:504] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:37.910727 23131 leader_election.cc:288] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:37.911410 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb056b49f4d44869749288abfd50d84" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:37.911438 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb056b49f4d44869749288abfd50d84" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:37.911643 21856 raft_consensus.cc:2454] T 8eb056b49f4d44869749288abfd50d84 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:37.911689 22120 raft_consensus.cc:2454] T 8eb056b49f4d44869749288abfd50d84 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:37.912094 22184 leader_election.cc:302] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:37.912245 21988 raft_consensus.cc:2454] T 5ccc8d73d1ea4539a170022fe6ad890e P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:37.912413 22251 raft_consensus.cc:2454] T 5ccc8d73d1ea4539a170022fe6ad890e P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:37.912418 23131 raft_consensus.cc:2781] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:37.912740 23131 raft_consensus.cc:482] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:37.912708 21790 leader_election.cc:302] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:53:37.912827 23131 raft_consensus.cc:3037] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.912990 23215 raft_consensus.cc:2781] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:37.913177 23215 raft_consensus.cc:686] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:53:37.913445 23215 consensus_queue.cc:237] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:37.915138 23131 raft_consensus.cc:504] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:37.916030 23131 leader_election.cc:288] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:37.916194 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb056b49f4d44869749288abfd50d84" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:37.916239 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb056b49f4d44869749288abfd50d84" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:37.916407 21856 raft_consensus.cc:3037] T 8eb056b49f4d44869749288abfd50d84 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.916451 22120 raft_consensus.cc:3037] T 8eb056b49f4d44869749288abfd50d84 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.916420 21711 catalog_manager.cc:5484] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:53:37.918702 21856 raft_consensus.cc:2454] T 8eb056b49f4d44869749288abfd50d84 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:37.918851 22120 raft_consensus.cc:2454] T 8eb056b49f4d44869749288abfd50d84 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:37.919263 22184 leader_election.cc:302] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:37.919553 23131 raft_consensus.cc:2781] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:37.919800 23131 raft_consensus.cc:686] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:53:37.920027 23131 consensus_queue.cc:237] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:37.923290 21711 catalog_manager.cc:5484] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:53:38.263279 23215 consensus_queue.cc:1035] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:38.271884 23215 consensus_queue.cc:1035] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:38.288635 23224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:38.289072 23224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:38.297529 23224 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:53:38.308091 23215 consensus_queue.cc:1035] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:38.315308 23098 consensus_queue.cc:1035] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240502 01:53:38.427695 23131 consensus_queue.cc:1035] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:38.453169 23131 consensus_queue.cc:1035] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 259.236 ms
time per row: 0.129618 ms
I20240502 01:53:38.735963 21707 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:38.747084 21968 tablet_service.cc:1467] Processing CreateTablet for tablet 0d246b118c1c4388ac04b3775ddbee89 (DEFAULT_TABLE table=rebalance_test_table_5 [id=868965c86d0d413fbbeaca41748259c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:38.747295 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 0d246b118c1c4388ac04b3775ddbee89 (DEFAULT_TABLE table=rebalance_test_table_5 [id=868965c86d0d413fbbeaca41748259c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:38.747670 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d246b118c1c4388ac04b3775ddbee89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:38.747778 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d246b118c1c4388ac04b3775ddbee89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:38.749202 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 3a27dc6af117475cb227e3526d2a9007 (DEFAULT_TABLE table=rebalance_test_table_5 [id=868965c86d0d413fbbeaca41748259c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:38.749714 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a27dc6af117475cb227e3526d2a9007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:38.749928 22099 tablet_service.cc:1467] Processing CreateTablet for tablet 766241df5ee24264a5cb7176e6c97b91 (DEFAULT_TABLE table=rebalance_test_table_5 [id=868965c86d0d413fbbeaca41748259c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:38.750298 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 766241df5ee24264a5cb7176e6c97b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:38.753413 22098 tablet_service.cc:1467] Processing CreateTablet for tablet 3a27dc6af117475cb227e3526d2a9007 (DEFAULT_TABLE table=rebalance_test_table_5 [id=868965c86d0d413fbbeaca41748259c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:38.753892 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a27dc6af117475cb227e3526d2a9007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:38.756706 22230 tablet_service.cc:1467] Processing CreateTablet for tablet 0d246b118c1c4388ac04b3775ddbee89 (DEFAULT_TABLE table=rebalance_test_table_5 [id=868965c86d0d413fbbeaca41748259c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:38.757169 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d246b118c1c4388ac04b3775ddbee89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:38.762341 21969 tablet_service.cc:1467] Processing CreateTablet for tablet 3a27dc6af117475cb227e3526d2a9007 (DEFAULT_TABLE table=rebalance_test_table_5 [id=868965c86d0d413fbbeaca41748259c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:38.762642 21835 tablet_service.cc:1467] Processing CreateTablet for tablet 766241df5ee24264a5cb7176e6c97b91 (DEFAULT_TABLE table=rebalance_test_table_5 [id=868965c86d0d413fbbeaca41748259c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:38.762817 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a27dc6af117475cb227e3526d2a9007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:38.763126 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 766241df5ee24264a5cb7176e6c97b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:38.765553 21967 tablet_service.cc:1467] Processing CreateTablet for tablet 766241df5ee24264a5cb7176e6c97b91 (DEFAULT_TABLE table=rebalance_test_table_5 [id=868965c86d0d413fbbeaca41748259c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:38.766072 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 766241df5ee24264a5cb7176e6c97b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:38.770422 23290 tablet_bootstrap.cc:492] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:38.774498 23291 tablet_bootstrap.cc:492] T 766241df5ee24264a5cb7176e6c97b91 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:38.775218 23290 tablet_bootstrap.cc:654] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:38.776801 23291 tablet_bootstrap.cc:654] T 766241df5ee24264a5cb7176e6c97b91 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:38.782452 23292 tablet_bootstrap.cc:492] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:38.785226 23292 tablet_bootstrap.cc:654] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:38.787077 23290 tablet_bootstrap.cc:492] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:38.787351 23290 ts_tablet_manager.cc:1367] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.010s
I20240502 01:53:38.787942 23292 tablet_bootstrap.cc:492] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:38.788177 23292 ts_tablet_manager.cc:1367] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240502 01:53:38.788321 23290 raft_consensus.cc:348] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:38.788619 23290 raft_consensus.cc:374] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:38.788780 23290 raft_consensus.cc:729] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:38.789116 23292 raft_consensus.cc:348] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:38.789163 23290 consensus_queue.cc:260] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:38.789774 23290 ts_tablet_manager.cc:1398] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:38.790591 23290 tablet_bootstrap.cc:492] T 0d246b118c1c4388ac04b3775ddbee89 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:38.790993 23291 tablet_bootstrap.cc:492] T 766241df5ee24264a5cb7176e6c97b91 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:38.791232 23291 ts_tablet_manager.cc:1367] T 766241df5ee24264a5cb7176e6c97b91 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240502 01:53:38.792136 23291 raft_consensus.cc:348] T 766241df5ee24264a5cb7176e6c97b91 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:38.792336 23290 tablet_bootstrap.cc:654] T 0d246b118c1c4388ac04b3775ddbee89 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:38.792464 23291 raft_consensus.cc:374] T 766241df5ee24264a5cb7176e6c97b91 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:38.792620 23291 raft_consensus.cc:729] T 766241df5ee24264a5cb7176e6c97b91 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:38.792958 23291 consensus_queue.cc:260] T 766241df5ee24264a5cb7176e6c97b91 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:38.796006 23291 ts_tablet_manager.cc:1398] T 766241df5ee24264a5cb7176e6c97b91 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:53:38.796500 23291 tablet_bootstrap.cc:492] T 3a27dc6af117475cb227e3526d2a9007 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:38.798035 23294 tablet_bootstrap.cc:492] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:38.798288 23292 raft_consensus.cc:374] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:38.798521 23292 raft_consensus.cc:729] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:38.800177 23294 tablet_bootstrap.cc:654] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:38.800824 23292 consensus_queue.cc:260] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:38.801198 23290 tablet_bootstrap.cc:492] T 0d246b118c1c4388ac04b3775ddbee89 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:38.801456 23290 ts_tablet_manager.cc:1367] T 0d246b118c1c4388ac04b3775ddbee89 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20240502 01:53:38.802656 23291 tablet_bootstrap.cc:654] T 3a27dc6af117475cb227e3526d2a9007 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:38.803402 23292 ts_tablet_manager.cc:1398] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 01:53:38.802814 23290 raft_consensus.cc:348] T 0d246b118c1c4388ac04b3775ddbee89 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:38.804056 23290 raft_consensus.cc:374] T 0d246b118c1c4388ac04b3775ddbee89 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:38.804200 23290 raft_consensus.cc:729] T 0d246b118c1c4388ac04b3775ddbee89 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:38.804572 23290 consensus_queue.cc:260] T 0d246b118c1c4388ac04b3775ddbee89 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:38.806993 23290 ts_tablet_manager.cc:1398] T 0d246b118c1c4388ac04b3775ddbee89 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:53:38.809098 23294 tablet_bootstrap.cc:492] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:38.809325 23294 ts_tablet_manager.cc:1367] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 01:53:38.810235 23294 raft_consensus.cc:348] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:38.810536 23294 raft_consensus.cc:374] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:38.810690 23294 raft_consensus.cc:729] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:38.811035 23294 consensus_queue.cc:260] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:38.811954 23294 ts_tablet_manager.cc:1398] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:38.812503 23294 tablet_bootstrap.cc:492] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:38.814599 23294 tablet_bootstrap.cc:654] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:38.822307 23294 tablet_bootstrap.cc:492] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:38.823120 23291 tablet_bootstrap.cc:492] T 3a27dc6af117475cb227e3526d2a9007 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:38.823405 23291 ts_tablet_manager.cc:1367] T 3a27dc6af117475cb227e3526d2a9007 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.000s
I20240502 01:53:38.824038 23274 raft_consensus.cc:482] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:38.824244 23291 raft_consensus.cc:348] T 3a27dc6af117475cb227e3526d2a9007 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:38.824268 23274 raft_consensus.cc:504] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:38.824469 23291 raft_consensus.cc:374] T 3a27dc6af117475cb227e3526d2a9007 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:38.824539 23291 raft_consensus.cc:729] T 3a27dc6af117475cb227e3526d2a9007 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:38.824813 23291 consensus_queue.cc:260] T 3a27dc6af117475cb227e3526d2a9007 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:38.825354 23291 ts_tablet_manager.cc:1398] T 3a27dc6af117475cb227e3526d2a9007 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:38.825410 23274 leader_election.cc:288] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:38.825677 23291 tablet_bootstrap.cc:492] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:38.827435 23294 ts_tablet_manager.cc:1367] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240502 01:53:38.827853 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "766241df5ee24264a5cb7176e6c97b91" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:38.828186 21989 raft_consensus.cc:2454] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:38.828253 23294 raft_consensus.cc:348] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:38.828497 23294 raft_consensus.cc:374] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:38.828577 23294 raft_consensus.cc:729] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:38.828828 23294 consensus_queue.cc:260] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:38.829033 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "766241df5ee24264a5cb7176e6c97b91" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:38.829294 22118 raft_consensus.cc:2454] T 766241df5ee24264a5cb7176e6c97b91 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:38.829393 23294 ts_tablet_manager.cc:1398] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:38.829696 23294 tablet_bootstrap.cc:492] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:38.830617 23291 tablet_bootstrap.cc:654] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:38.833577 23294 tablet_bootstrap.cc:654] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:38.833496 21790 leader_election.cc:302] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:53:38.836719 23274 raft_consensus.cc:2781] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:38.836912 23274 raft_consensus.cc:482] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:38.837035 23274 raft_consensus.cc:3037] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:38.841383 23274 raft_consensus.cc:504] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:38.842883 23294 tablet_bootstrap.cc:492] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:38.843021 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "766241df5ee24264a5cb7176e6c97b91" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:38.843142 23294 ts_tablet_manager.cc:1367] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20240502 01:53:38.843338 21989 raft_consensus.cc:3037] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:38.843669 23291 tablet_bootstrap.cc:492] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:38.843896 23291 ts_tablet_manager.cc:1367] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20240502 01:53:38.844326 23294 raft_consensus.cc:348] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:38.844717 23294 raft_consensus.cc:374] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:38.844920 23294 raft_consensus.cc:729] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:38.844796 23291 raft_consensus.cc:348] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:38.845024 23291 raft_consensus.cc:374] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:38.845161 23291 raft_consensus.cc:729] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:38.845386 23294 consensus_queue.cc:260] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:38.846060 23294 ts_tablet_manager.cc:1398] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:38.846458 23291 consensus_queue.cc:260] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:38.847802 23291 ts_tablet_manager.cc:1398] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:38.849671 21989 raft_consensus.cc:2454] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:38.850328 21790 leader_election.cc:302] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:53:38.850716 23215 raft_consensus.cc:2781] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:38.850899 23215 raft_consensus.cc:686] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:53:38.851238 23215 consensus_queue.cc:237] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:38.857882 23140 raft_consensus.cc:482] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:38.857939 21707 catalog_manager.cc:5484] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:53:38.858086 23140 raft_consensus.cc:504] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:38.859200 23140 leader_election.cc:288] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:38.862803 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d246b118c1c4388ac04b3775ddbee89" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:38.863119 22118 raft_consensus.cc:2454] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:38.866626 21924 leader_election.cc:302] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:38.867197 23140 raft_consensus.cc:2781] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:38.867341 23140 raft_consensus.cc:482] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:38.867472 23140 raft_consensus.cc:3037] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:38.871498 23274 leader_election.cc:288] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:38.871841 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d246b118c1c4388ac04b3775ddbee89" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:38.872123 22251 raft_consensus.cc:2454] T 0d246b118c1c4388ac04b3775ddbee89 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:38.872912 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "766241df5ee24264a5cb7176e6c97b91" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:38.873180 22118 raft_consensus.cc:3037] T 766241df5ee24264a5cb7176e6c97b91 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:38.877091 22118 raft_consensus.cc:2454] T 766241df5ee24264a5cb7176e6c97b91 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:38.878473 23140 raft_consensus.cc:504] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:38.879330 23140 leader_election.cc:288] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:38.879863 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d246b118c1c4388ac04b3775ddbee89" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:38.880103 22118 raft_consensus.cc:3037] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:38.880331 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d246b118c1c4388ac04b3775ddbee89" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:38.880589 22251 raft_consensus.cc:3037] T 0d246b118c1c4388ac04b3775ddbee89 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:38.883052 22118 raft_consensus.cc:2454] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:38.883427 22251 raft_consensus.cc:2454] T 0d246b118c1c4388ac04b3775ddbee89 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:38.884410 21924 leader_election.cc:302] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:38.884781 23140 raft_consensus.cc:2781] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:38.885552 23140 raft_consensus.cc:686] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:53:38.886032 23140 consensus_queue.cc:237] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:38.889847 21707 catalog_manager.cc:5484] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:53:39.040176 23266 raft_consensus.cc:482] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:39.040375 23266 raft_consensus.cc:504] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:39.041107 23266 leader_election.cc:288] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:39.041646 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a27dc6af117475cb227e3526d2a9007" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:39.041747 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a27dc6af117475cb227e3526d2a9007" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:39.041884 21989 raft_consensus.cc:2454] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:39.042011 22118 raft_consensus.cc:2454] T 3a27dc6af117475cb227e3526d2a9007 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:39.042353 22184 leader_election.cc:302] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:39.042652 23266 raft_consensus.cc:2781] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:39.042759 23266 raft_consensus.cc:482] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:39.042827 23266 raft_consensus.cc:3037] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:39.044795 23266 raft_consensus.cc:504] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:39.045413 23266 leader_election.cc:288] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:39.045966 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a27dc6af117475cb227e3526d2a9007" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:39.045945 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a27dc6af117475cb227e3526d2a9007" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:39.046177 22118 raft_consensus.cc:3037] T 3a27dc6af117475cb227e3526d2a9007 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:39.046207 21989 raft_consensus.cc:3037] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:39.049113 22118 raft_consensus.cc:2454] T 3a27dc6af117475cb227e3526d2a9007 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:39.049095 21989 raft_consensus.cc:2454] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:39.049846 22184 leader_election.cc:302] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:39.050180 23266 raft_consensus.cc:2781] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:39.050326 23266 raft_consensus.cc:686] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:53:39.050602 23266 consensus_queue.cc:237] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:39.053543 21707 catalog_manager.cc:5484] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:53:39.278865 23274 consensus_queue.cc:1035] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:39.303799 23274 consensus_queue.cc:1035] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:39.447441 23267 consensus_queue.cc:1035] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:39.453613 23140 consensus_queue.cc:1035] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:39.459898 23308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:39.460354 23308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:53:39.466976 23140 consensus_queue.cc:1035] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:39.469450 23308 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:53:39.477121 23323 consensus_queue.cc:1035] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
INSERT report
rows total: 2000
time total: 230.693 ms
time per row: 0.115346 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:53:40.825233 21709 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:40.835287 21835 tablet_service.cc:1467] Processing CreateTablet for tablet 21a05c0b257f44708fb0eac9540f2218 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ab3326b89b7f41eab7e16c19f0f25190]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:40.835407 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 21a05c0b257f44708fb0eac9540f2218 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ab3326b89b7f41eab7e16c19f0f25190]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:40.835821 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a05c0b257f44708fb0eac9540f2218. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:40.835875 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a05c0b257f44708fb0eac9540f2218. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:40.836652 21967 tablet_service.cc:1467] Processing CreateTablet for tablet 21a05c0b257f44708fb0eac9540f2218 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ab3326b89b7f41eab7e16c19f0f25190]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:40.837093 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a05c0b257f44708fb0eac9540f2218. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:40.837260 21968 tablet_service.cc:1467] Processing CreateTablet for tablet 484da49a539a47f199b182041cb659bd (DEFAULT_TABLE table=rebalance_test_table_6 [id=ab3326b89b7f41eab7e16c19f0f25190]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:40.837536 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 484da49a539a47f199b182041cb659bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:40.839766 22099 tablet_service.cc:1467] Processing CreateTablet for tablet 51312aa1f74c43588df6facab76ea1c6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ab3326b89b7f41eab7e16c19f0f25190]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:40.840209 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51312aa1f74c43588df6facab76ea1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:40.841749 22229 tablet_service.cc:1467] Processing CreateTablet for tablet 484da49a539a47f199b182041cb659bd (DEFAULT_TABLE table=rebalance_test_table_6 [id=ab3326b89b7f41eab7e16c19f0f25190]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:40.842172 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 484da49a539a47f199b182041cb659bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:40.844040 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 51312aa1f74c43588df6facab76ea1c6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ab3326b89b7f41eab7e16c19f0f25190]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:40.844511 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51312aa1f74c43588df6facab76ea1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:40.841719 22230 tablet_service.cc:1467] Processing CreateTablet for tablet 51312aa1f74c43588df6facab76ea1c6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ab3326b89b7f41eab7e16c19f0f25190]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:40.845212 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51312aa1f74c43588df6facab76ea1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:40.849342 21837 tablet_service.cc:1467] Processing CreateTablet for tablet 484da49a539a47f199b182041cb659bd (DEFAULT_TABLE table=rebalance_test_table_6 [id=ab3326b89b7f41eab7e16c19f0f25190]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:40.849753 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 484da49a539a47f199b182041cb659bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:40.854789 23377 tablet_bootstrap.cc:492] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:40.856601 23377 tablet_bootstrap.cc:654] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:40.860288 23375 tablet_bootstrap.cc:492] T 484da49a539a47f199b182041cb659bd P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:40.861455 23377 tablet_bootstrap.cc:492] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:40.861725 23377 ts_tablet_manager.cc:1367] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 01:53:40.861941 23378 tablet_bootstrap.cc:492] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:40.862474 23375 tablet_bootstrap.cc:654] T 484da49a539a47f199b182041cb659bd P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:40.862660 23377 raft_consensus.cc:348] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:40.862988 23377 raft_consensus.cc:374] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:40.862993 23376 tablet_bootstrap.cc:492] T 51312aa1f74c43588df6facab76ea1c6 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:40.863175 23377 raft_consensus.cc:729] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:40.863509 23377 consensus_queue.cc:260] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:40.864610 23377 ts_tablet_manager.cc:1398] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:40.864946 23378 tablet_bootstrap.cc:654] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:40.866341 23376 tablet_bootstrap.cc:654] T 51312aa1f74c43588df6facab76ea1c6 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:40.870556 23375 tablet_bootstrap.cc:492] T 484da49a539a47f199b182041cb659bd P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:40.870837 23375 ts_tablet_manager.cc:1367] T 484da49a539a47f199b182041cb659bd P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 01:53:40.871682 23375 raft_consensus.cc:348] T 484da49a539a47f199b182041cb659bd P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:40.871892 23375 raft_consensus.cc:374] T 484da49a539a47f199b182041cb659bd P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:40.872027 23375 raft_consensus.cc:729] T 484da49a539a47f199b182041cb659bd P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:40.872308 23375 consensus_queue.cc:260] T 484da49a539a47f199b182041cb659bd P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:40.873023 23375 ts_tablet_manager.cc:1398] T 484da49a539a47f199b182041cb659bd P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:40.873463 23376 tablet_bootstrap.cc:492] T 51312aa1f74c43588df6facab76ea1c6 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:40.873661 23375 tablet_bootstrap.cc:492] T 21a05c0b257f44708fb0eac9540f2218 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:40.873732 23376 ts_tablet_manager.cc:1367] T 51312aa1f74c43588df6facab76ea1c6 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20240502 01:53:40.874989 23376 raft_consensus.cc:348] T 51312aa1f74c43588df6facab76ea1c6 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:40.875365 23376 raft_consensus.cc:374] T 51312aa1f74c43588df6facab76ea1c6 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:40.875532 23376 raft_consensus.cc:729] T 51312aa1f74c43588df6facab76ea1c6 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:40.875916 23376 consensus_queue.cc:260] T 51312aa1f74c43588df6facab76ea1c6 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:40.877468 23375 tablet_bootstrap.cc:654] T 21a05c0b257f44708fb0eac9540f2218 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:40.877878 23376 ts_tablet_manager.cc:1398] T 51312aa1f74c43588df6facab76ea1c6 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:53:40.878340 23376 tablet_bootstrap.cc:492] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:40.881867 23376 tablet_bootstrap.cc:654] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:40.882320 23378 tablet_bootstrap.cc:492] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:40.882571 23378 ts_tablet_manager.cc:1367] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.012s
I20240502 01:53:40.883988 23378 raft_consensus.cc:348] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:40.884223 23375 tablet_bootstrap.cc:492] T 21a05c0b257f44708fb0eac9540f2218 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:40.884289 23378 raft_consensus.cc:374] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:40.884398 23375 ts_tablet_manager.cc:1367] T 21a05c0b257f44708fb0eac9540f2218 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20240502 01:53:40.884414 23378 raft_consensus.cc:729] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:40.884807 23378 consensus_queue.cc:260] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:40.885151 23375 raft_consensus.cc:348] T 21a05c0b257f44708fb0eac9540f2218 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:40.885452 23375 raft_consensus.cc:374] T 21a05c0b257f44708fb0eac9540f2218 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:40.885579 23375 raft_consensus.cc:729] T 21a05c0b257f44708fb0eac9540f2218 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:40.885669 23378 ts_tablet_manager.cc:1398] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:40.885974 23375 consensus_queue.cc:260] T 21a05c0b257f44708fb0eac9540f2218 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:40.886122 23378 tablet_bootstrap.cc:492] T 51312aa1f74c43588df6facab76ea1c6 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:40.886790 23375 ts_tablet_manager.cc:1398] T 21a05c0b257f44708fb0eac9540f2218 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:40.888337 23378 tablet_bootstrap.cc:654] T 51312aa1f74c43588df6facab76ea1c6 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:40.890125 23376 tablet_bootstrap.cc:492] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:40.890450 23376 ts_tablet_manager.cc:1367] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20240502 01:53:40.891513 23376 raft_consensus.cc:348] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:40.891983 23376 raft_consensus.cc:374] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:40.892120 23376 raft_consensus.cc:729] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:40.892467 23376 consensus_queue.cc:260] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:40.893265 23376 ts_tablet_manager.cc:1398] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:53:40.893591 23378 tablet_bootstrap.cc:492] T 51312aa1f74c43588df6facab76ea1c6 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:40.893608 23376 tablet_bootstrap.cc:492] T 21a05c0b257f44708fb0eac9540f2218 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:40.893783 23378 ts_tablet_manager.cc:1367] T 51312aa1f74c43588df6facab76ea1c6 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20240502 01:53:40.894553 23378 raft_consensus.cc:348] T 51312aa1f74c43588df6facab76ea1c6 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:40.894817 23378 raft_consensus.cc:374] T 51312aa1f74c43588df6facab76ea1c6 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:40.894891 23378 raft_consensus.cc:729] T 51312aa1f74c43588df6facab76ea1c6 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:40.895169 23378 consensus_queue.cc:260] T 51312aa1f74c43588df6facab76ea1c6 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:40.895377 23376 tablet_bootstrap.cc:654] T 21a05c0b257f44708fb0eac9540f2218 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:40.895826 23378 ts_tablet_manager.cc:1398] T 51312aa1f74c43588df6facab76ea1c6 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:40.896173 23378 tablet_bootstrap.cc:492] T 484da49a539a47f199b182041cb659bd P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:40.898548 23378 tablet_bootstrap.cc:654] T 484da49a539a47f199b182041cb659bd P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:40.902153 23376 tablet_bootstrap.cc:492] T 21a05c0b257f44708fb0eac9540f2218 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:40.902268 23378 tablet_bootstrap.cc:492] T 484da49a539a47f199b182041cb659bd P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:40.902309 23376 ts_tablet_manager.cc:1367] T 21a05c0b257f44708fb0eac9540f2218 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20240502 01:53:40.902451 23378 ts_tablet_manager.cc:1367] T 484da49a539a47f199b182041cb659bd P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20240502 01:53:40.903069 23376 raft_consensus.cc:348] T 21a05c0b257f44708fb0eac9540f2218 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:40.903242 23376 raft_consensus.cc:374] T 21a05c0b257f44708fb0eac9540f2218 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:40.903367 23376 raft_consensus.cc:729] T 21a05c0b257f44708fb0eac9540f2218 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:40.903312 23378 raft_consensus.cc:348] T 484da49a539a47f199b182041cb659bd P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:40.903592 23378 raft_consensus.cc:374] T 484da49a539a47f199b182041cb659bd P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:40.903717 23378 raft_consensus.cc:729] T 484da49a539a47f199b182041cb659bd P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:40.903618 23376 consensus_queue.cc:260] T 21a05c0b257f44708fb0eac9540f2218 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:40.903954 23378 consensus_queue.cc:260] T 484da49a539a47f199b182041cb659bd P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:40.904197 23376 ts_tablet_manager.cc:1398] T 21a05c0b257f44708fb0eac9540f2218 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:40.904572 23378 ts_tablet_manager.cc:1398] T 484da49a539a47f199b182041cb659bd P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:40.961764 23370 raft_consensus.cc:482] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:40.961989 23370 raft_consensus.cc:504] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:40.962785 23370 leader_election.cc:288] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:40.963347 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a05c0b257f44708fb0eac9540f2218" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:40.963346 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a05c0b257f44708fb0eac9540f2218" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:40.963614 22251 raft_consensus.cc:2454] T 21a05c0b257f44708fb0eac9540f2218 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:40.963634 21988 raft_consensus.cc:2454] T 21a05c0b257f44708fb0eac9540f2218 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:40.964124 21792 leader_election.cc:302] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:40.964505 23366 raft_consensus.cc:2781] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:40.964625 23366 raft_consensus.cc:482] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:40.964761 23366 raft_consensus.cc:3037] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:40.967202 23366 raft_consensus.cc:504] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:40.967784 23366 leader_election.cc:288] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:40.968231 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a05c0b257f44708fb0eac9540f2218" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:40.968247 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a05c0b257f44708fb0eac9540f2218" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:40.968416 22251 raft_consensus.cc:3037] T 21a05c0b257f44708fb0eac9540f2218 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:40.968443 21988 raft_consensus.cc:3037] T 21a05c0b257f44708fb0eac9540f2218 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:40.970492 22251 raft_consensus.cc:2454] T 21a05c0b257f44708fb0eac9540f2218 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:40.970499 21988 raft_consensus.cc:2454] T 21a05c0b257f44708fb0eac9540f2218 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:40.970914 21790 leader_election.cc:302] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:40.971246 23366 raft_consensus.cc:2781] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:40.971401 23366 raft_consensus.cc:686] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:53:40.971719 23366 consensus_queue.cc:237] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:40.974771 21707 catalog_manager.cc:5484] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:53:41.021155 23266 raft_consensus.cc:482] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:41.021392 23266 raft_consensus.cc:504] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:41.022086 23266 leader_election.cc:288] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:41.022773 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "484da49a539a47f199b182041cb659bd" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:41.022802 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "484da49a539a47f199b182041cb659bd" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:41.022990 21857 raft_consensus.cc:2454] T 484da49a539a47f199b182041cb659bd P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:41.023027 21988 raft_consensus.cc:2454] T 484da49a539a47f199b182041cb659bd P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:41.023429 22184 leader_election.cc:302] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:41.023734 23266 raft_consensus.cc:2781] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:41.023835 23266 raft_consensus.cc:482] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:41.023945 23266 raft_consensus.cc:3037] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:41.025926 23266 raft_consensus.cc:504] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:41.026548 23266 leader_election.cc:288] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:41.026999 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "484da49a539a47f199b182041cb659bd" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:41.027065 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "484da49a539a47f199b182041cb659bd" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:41.027225 21857 raft_consensus.cc:3037] T 484da49a539a47f199b182041cb659bd P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:41.027283 21988 raft_consensus.cc:3037] T 484da49a539a47f199b182041cb659bd P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:41.029271 21857 raft_consensus.cc:2454] T 484da49a539a47f199b182041cb659bd P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:41.029425 21988 raft_consensus.cc:2454] T 484da49a539a47f199b182041cb659bd P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:41.029783 22184 leader_election.cc:302] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:41.030082 23266 raft_consensus.cc:2781] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:41.030506 23266 raft_consensus.cc:686] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:53:41.030762 23266 consensus_queue.cc:237] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:41.033461 21707 catalog_manager.cc:5484] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:53:41.157228 23314 raft_consensus.cc:482] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:41.157438 23314 raft_consensus.cc:504] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:41.158221 23314 leader_election.cc:288] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:41.158797 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51312aa1f74c43588df6facab76ea1c6" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:41.158839 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51312aa1f74c43588df6facab76ea1c6" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:41.159049 22251 raft_consensus.cc:2454] T 51312aa1f74c43588df6facab76ea1c6 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:41.159121 21857 raft_consensus.cc:2454] T 51312aa1f74c43588df6facab76ea1c6 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:41.159533 22055 leader_election.cc:302] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:41.159839 23314 raft_consensus.cc:2781] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:41.159941 23314 raft_consensus.cc:482] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:41.160028 23314 raft_consensus.cc:3037] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:41.162338 23314 raft_consensus.cc:504] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:41.163057 23314 leader_election.cc:288] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:41.163532 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51312aa1f74c43588df6facab76ea1c6" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:41.163635 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51312aa1f74c43588df6facab76ea1c6" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:41.163811 22251 raft_consensus.cc:3037] T 51312aa1f74c43588df6facab76ea1c6 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:41.163879 21857 raft_consensus.cc:3037] T 51312aa1f74c43588df6facab76ea1c6 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:41.166793 22251 raft_consensus.cc:2454] T 51312aa1f74c43588df6facab76ea1c6 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:41.166801 21857 raft_consensus.cc:2454] T 51312aa1f74c43588df6facab76ea1c6 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:41.167269 22053 leader_election.cc:302] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:41.167763 23314 raft_consensus.cc:2781] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:41.168126 23314 raft_consensus.cc:686] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:53:41.168427 23314 consensus_queue.cc:237] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:41.171432 21707 catalog_manager.cc:5484] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:53:41.470652 23366 consensus_queue.cc:1035] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:41.478294 23366 consensus_queue.cc:1035] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:41.508122 23266 consensus_queue.cc:1035] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:41.513412 23266 consensus_queue.cc:1035] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:41.564013 23314 consensus_queue.cc:1035] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:41.575348 23314 consensus_queue.cc:1035] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:41.604758 23394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:41.605245 23394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:41.613567 23394 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
INSERT report
rows total: 2000
time total: 260.931 ms
time per row: 0.130465 ms
I20240502 01:53:42.025110 21707 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:42.035228 21967 tablet_service.cc:1467] Processing CreateTablet for tablet ec67f45b2b604c129d8572c5ca6acbc4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=cd297aa2796f428db6b815990620e2a4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:42.035460 22099 tablet_service.cc:1467] Processing CreateTablet for tablet ec67f45b2b604c129d8572c5ca6acbc4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=cd297aa2796f428db6b815990620e2a4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:42.035794 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec67f45b2b604c129d8572c5ca6acbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:42.035781 22100 tablet_service.cc:1467] Processing CreateTablet for tablet ba39439749a9486799373e173110f5c5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=cd297aa2796f428db6b815990620e2a4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:42.035964 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec67f45b2b604c129d8572c5ca6acbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:42.036155 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba39439749a9486799373e173110f5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:42.037292 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 5d40847a56704b33b771c3fd87d6dfae (DEFAULT_TABLE table=rebalance_test_table_7 [id=cd297aa2796f428db6b815990620e2a4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:42.037751 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d40847a56704b33b771c3fd87d6dfae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:42.040809 22230 tablet_service.cc:1467] Processing CreateTablet for tablet ba39439749a9486799373e173110f5c5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=cd297aa2796f428db6b815990620e2a4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:42.041479 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba39439749a9486799373e173110f5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:42.044924 22098 tablet_service.cc:1467] Processing CreateTablet for tablet 5d40847a56704b33b771c3fd87d6dfae (DEFAULT_TABLE table=rebalance_test_table_7 [id=cd297aa2796f428db6b815990620e2a4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:42.045603 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d40847a56704b33b771c3fd87d6dfae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:42.050606 21968 tablet_service.cc:1467] Processing CreateTablet for tablet ba39439749a9486799373e173110f5c5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=cd297aa2796f428db6b815990620e2a4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:42.051105 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba39439749a9486799373e173110f5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:42.051031 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 5d40847a56704b33b771c3fd87d6dfae (DEFAULT_TABLE table=rebalance_test_table_7 [id=cd297aa2796f428db6b815990620e2a4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:42.051445 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d40847a56704b33b771c3fd87d6dfae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:42.051504 23459 tablet_bootstrap.cc:492] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:42.053236 21835 tablet_service.cc:1467] Processing CreateTablet for tablet ec67f45b2b604c129d8572c5ca6acbc4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=cd297aa2796f428db6b815990620e2a4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:42.053629 23459 tablet_bootstrap.cc:654] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:42.053678 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec67f45b2b604c129d8572c5ca6acbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:42.055155 23458 tablet_bootstrap.cc:492] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:42.055709 23457 tablet_bootstrap.cc:492] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:42.057178 23458 tablet_bootstrap.cc:654] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:42.059168 23457 tablet_bootstrap.cc:654] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:42.059664 23458 tablet_bootstrap.cc:492] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:42.059839 23458 ts_tablet_manager.cc:1367] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:53:42.060748 23458 raft_consensus.cc:348] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:42.061309 23458 raft_consensus.cc:374] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:42.061322 23459 tablet_bootstrap.cc:492] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:42.061494 23459 ts_tablet_manager.cc:1367] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 01:53:42.061492 23458 raft_consensus.cc:729] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:42.061537 23457 tablet_bootstrap.cc:492] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:42.061687 23457 ts_tablet_manager.cc:1367] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240502 01:53:42.061766 23458 consensus_queue.cc:260] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:42.062714 23458 ts_tablet_manager.cc:1398] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:53:42.062870 23459 raft_consensus.cc:348] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:42.063210 23459 raft_consensus.cc:374] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:42.063371 23459 raft_consensus.cc:729] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:42.063400 23457 raft_consensus.cc:348] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:42.063035 23458 tablet_bootstrap.cc:492] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:42.063738 23457 raft_consensus.cc:374] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:42.063685 23459 consensus_queue.cc:260] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:42.063925 23457 raft_consensus.cc:729] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:42.064397 23459 ts_tablet_manager.cc:1398] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:42.065896 23457 consensus_queue.cc:260] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:42.068048 23457 ts_tablet_manager.cc:1398] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:53:42.068439 23457 tablet_bootstrap.cc:492] T ba39439749a9486799373e173110f5c5 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:42.070397 23457 tablet_bootstrap.cc:654] T ba39439749a9486799373e173110f5c5 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:42.077015 23458 tablet_bootstrap.cc:654] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:42.078240 23457 tablet_bootstrap.cc:492] T ba39439749a9486799373e173110f5c5 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:42.078429 23457 ts_tablet_manager.cc:1367] T ba39439749a9486799373e173110f5c5 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240502 01:53:42.079028 23459 tablet_bootstrap.cc:492] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:42.079337 23457 raft_consensus.cc:348] T ba39439749a9486799373e173110f5c5 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:42.079599 23457 raft_consensus.cc:374] T ba39439749a9486799373e173110f5c5 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:42.079699 23457 raft_consensus.cc:729] T ba39439749a9486799373e173110f5c5 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:42.079995 23457 consensus_queue.cc:260] T ba39439749a9486799373e173110f5c5 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:42.080580 23457 ts_tablet_manager.cc:1398] T ba39439749a9486799373e173110f5c5 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:42.080920 23457 tablet_bootstrap.cc:492] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:42.081298 23459 tablet_bootstrap.cc:654] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:42.082232 23458 tablet_bootstrap.cc:492] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:42.082422 23458 ts_tablet_manager.cc:1367] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240502 01:53:42.082504 23457 tablet_bootstrap.cc:654] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:42.082756 23464 tablet_bootstrap.cc:492] T 5d40847a56704b33b771c3fd87d6dfae P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:42.084686 23464 tablet_bootstrap.cc:654] T 5d40847a56704b33b771c3fd87d6dfae P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:42.084322 23458 raft_consensus.cc:348] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:42.085196 23458 raft_consensus.cc:374] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:42.085304 23458 raft_consensus.cc:729] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:42.085740 23458 consensus_queue.cc:260] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:42.088899 23458 ts_tablet_manager.cc:1398] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:53:42.090487 23464 tablet_bootstrap.cc:492] T 5d40847a56704b33b771c3fd87d6dfae P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:42.090659 23464 ts_tablet_manager.cc:1367] T 5d40847a56704b33b771c3fd87d6dfae P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240502 01:53:42.091452 23464 raft_consensus.cc:348] T 5d40847a56704b33b771c3fd87d6dfae P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:42.091676 23464 raft_consensus.cc:374] T 5d40847a56704b33b771c3fd87d6dfae P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:42.091769 23464 raft_consensus.cc:729] T 5d40847a56704b33b771c3fd87d6dfae P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:42.092060 23464 consensus_queue.cc:260] T 5d40847a56704b33b771c3fd87d6dfae P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:42.093097 23464 ts_tablet_manager.cc:1398] T 5d40847a56704b33b771c3fd87d6dfae P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:42.093416 23464 tablet_bootstrap.cc:492] T ec67f45b2b604c129d8572c5ca6acbc4 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:42.094785 23459 tablet_bootstrap.cc:492] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:42.095021 23459 ts_tablet_manager.cc:1367] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240502 01:53:42.095182 23464 tablet_bootstrap.cc:654] T ec67f45b2b604c129d8572c5ca6acbc4 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:42.096074 23459 raft_consensus.cc:348] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:42.096446 23459 raft_consensus.cc:374] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:42.096534 23459 raft_consensus.cc:729] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:42.096897 23459 consensus_queue.cc:260] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:42.100920 23459 ts_tablet_manager.cc:1398] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
I20240502 01:53:42.104607 23457 tablet_bootstrap.cc:492] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:42.104804 23457 ts_tablet_manager.cc:1367] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.015s
I20240502 01:53:42.105584 23457 raft_consensus.cc:348] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:42.105769 23457 raft_consensus.cc:374] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:42.105870 23457 raft_consensus.cc:729] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:42.106109 23457 consensus_queue.cc:260] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:42.106693 23457 ts_tablet_manager.cc:1398] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:42.107852 23464 tablet_bootstrap.cc:492] T ec67f45b2b604c129d8572c5ca6acbc4 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:42.108072 23464 ts_tablet_manager.cc:1367] T ec67f45b2b604c129d8572c5ca6acbc4 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20240502 01:53:42.109072 23464 raft_consensus.cc:348] T ec67f45b2b604c129d8572c5ca6acbc4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:42.109282 23464 raft_consensus.cc:374] T ec67f45b2b604c129d8572c5ca6acbc4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:42.109359 23464 raft_consensus.cc:729] T ec67f45b2b604c129d8572c5ca6acbc4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:42.109652 23464 consensus_queue.cc:260] T ec67f45b2b604c129d8572c5ca6acbc4 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:42.110252 23464 ts_tablet_manager.cc:1398] T ec67f45b2b604c129d8572c5ca6acbc4 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:42.116262 23390 raft_consensus.cc:482] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:42.116444 23390 raft_consensus.cc:504] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:42.117250 23390 leader_election.cc:288] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:42.117784 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40847a56704b33b771c3fd87d6dfae" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:42.117843 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40847a56704b33b771c3fd87d6dfae" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:42.118065 21856 raft_consensus.cc:2454] T 5d40847a56704b33b771c3fd87d6dfae P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:42.118086 22119 raft_consensus.cc:2454] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:42.118536 22186 leader_election.cc:302] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:42.118817 23390 raft_consensus.cc:2781] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:42.118937 23390 raft_consensus.cc:482] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:42.119024 23390 raft_consensus.cc:3037] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:42.121054 23390 raft_consensus.cc:504] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:42.121721 23390 leader_election.cc:288] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:42.122246 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40847a56704b33b771c3fd87d6dfae" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:42.122515 22119 raft_consensus.cc:3037] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:42.124488 22119 raft_consensus.cc:2454] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:42.124994 22186 leader_election.cc:302] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:42.124975 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40847a56704b33b771c3fd87d6dfae" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:42.125186 21856 raft_consensus.cc:3037] T 5d40847a56704b33b771c3fd87d6dfae P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:42.125293 23390 raft_consensus.cc:2781] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:42.125440 23390 raft_consensus.cc:686] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:53:42.125723 23390 consensus_queue.cc:237] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:42.128064 21856 raft_consensus.cc:2454] T 5d40847a56704b33b771c3fd87d6dfae P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:42.129065 21709 catalog_manager.cc:5484] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:53:42.192468 23390 raft_consensus.cc:482] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:42.192718 23390 raft_consensus.cc:504] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:42.193627 23390 leader_election.cc:288] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:42.194268 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba39439749a9486799373e173110f5c5" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:42.194268 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba39439749a9486799373e173110f5c5" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:42.194548 21989 raft_consensus.cc:2454] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:42.194548 22119 raft_consensus.cc:2454] T ba39439749a9486799373e173110f5c5 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:42.195089 22184 leader_election.cc:302] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:42.195472 23390 raft_consensus.cc:2781] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:42.195664 23390 raft_consensus.cc:482] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:42.195811 23390 raft_consensus.cc:3037] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:42.197890 23390 raft_consensus.cc:504] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:42.198498 23390 leader_election.cc:288] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:42.199014 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba39439749a9486799373e173110f5c5" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:42.199012 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba39439749a9486799373e173110f5c5" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:42.199213 22119 raft_consensus.cc:3037] T ba39439749a9486799373e173110f5c5 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:42.199208 21989 raft_consensus.cc:3037] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:42.201225 21989 raft_consensus.cc:2454] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:42.201297 22119 raft_consensus.cc:2454] T ba39439749a9486799373e173110f5c5 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:42.201761 22184 leader_election.cc:302] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:42.202045 23390 raft_consensus.cc:2781] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:42.202239 23390 raft_consensus.cc:686] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:53:42.202565 23321 raft_consensus.cc:482] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:42.202538 23390 consensus_queue.cc:237] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:42.202718 23321 raft_consensus.cc:504] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:42.203526 23321 leader_election.cc:288] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:42.204468 21855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec67f45b2b604c129d8572c5ca6acbc4" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:42.204799 21855 raft_consensus.cc:2454] T ec67f45b2b604c129d8572c5ca6acbc4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:42.205461 21922 leader_election.cc:302] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:53:42.205780 23321 raft_consensus.cc:2781] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:42.205905 23321 raft_consensus.cc:482] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:42.205997 23321 raft_consensus.cc:3037] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:42.206002 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec67f45b2b604c129d8572c5ca6acbc4" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:42.206244 22119 raft_consensus.cc:2454] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:42.208431 23321 raft_consensus.cc:504] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:42.209002 23321 leader_election.cc:288] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:42.209517 21855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec67f45b2b604c129d8572c5ca6acbc4" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:42.209834 21855 raft_consensus.cc:3037] T ec67f45b2b604c129d8572c5ca6acbc4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:42.209982 21709 catalog_manager.cc:5484] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:53:42.212093 21855 raft_consensus.cc:2454] T ec67f45b2b604c129d8572c5ca6acbc4 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:42.212642 21922 leader_election.cc:302] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:53:42.212966 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec67f45b2b604c129d8572c5ca6acbc4" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:42.213209 22119 raft_consensus.cc:3037] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:42.215359 22119 raft_consensus.cc:2454] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:42.215691 23321 raft_consensus.cc:2781] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:42.215843 23321 raft_consensus.cc:686] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:53:42.216085 23321 consensus_queue.cc:237] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:42.218969 21709 catalog_manager.cc:5484] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:53:42.564390 23390 consensus_queue.cc:1035] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:42.580839 23390 consensus_queue.cc:1035] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:42.604223 23474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:42.604647 23474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:42.613294 23474 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 01:53:42.685212 23471 consensus_queue.cc:1035] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:42.705734 23321 consensus_queue.cc:1035] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:42.760026 23390 consensus_queue.cc:1035] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:42.784739 23402 consensus_queue.cc:1035] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:42.823041 23513 mvcc.cc:205] Tried to move back new op lower bound from 7023062314190331904 to 7023062311744634880. Current Snapshot: MvccSnapshot[applied={T|T < 7023062314190331904}]
INSERT report
rows total: 2000
time total: 300.109 ms
time per row: 0.150055 ms
I20240502 01:53:43.070505 21709 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:43.079661 21968 tablet_service.cc:1467] Processing CreateTablet for tablet e60a8beee9284c98ae6193af2c58e8c7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=69aebb264a8d4fbebe1a1e02330fb403]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:43.079946 21836 tablet_service.cc:1467] Processing CreateTablet for tablet e60a8beee9284c98ae6193af2c58e8c7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=69aebb264a8d4fbebe1a1e02330fb403]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:43.080171 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e60a8beee9284c98ae6193af2c58e8c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:43.080404 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e60a8beee9284c98ae6193af2c58e8c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:43.081604 22099 tablet_service.cc:1467] Processing CreateTablet for tablet 8421b561a3a2402bafee202c129f4c14 (DEFAULT_TABLE table=rebalance_test_table_8 [id=69aebb264a8d4fbebe1a1e02330fb403]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:43.082023 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8421b561a3a2402bafee202c129f4c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:43.082041 22230 tablet_service.cc:1467] Processing CreateTablet for tablet 8421b561a3a2402bafee202c129f4c14 (DEFAULT_TABLE table=rebalance_test_table_8 [id=69aebb264a8d4fbebe1a1e02330fb403]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:43.082101 21835 tablet_service.cc:1467] Processing CreateTablet for tablet ab581c1e0054477c849242f57b5066f0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=69aebb264a8d4fbebe1a1e02330fb403]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:43.082473 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8421b561a3a2402bafee202c129f4c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:43.082747 22231 tablet_service.cc:1467] Processing CreateTablet for tablet ab581c1e0054477c849242f57b5066f0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=69aebb264a8d4fbebe1a1e02330fb403]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:43.083123 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab581c1e0054477c849242f57b5066f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:43.081565 22098 tablet_service.cc:1467] Processing CreateTablet for tablet e60a8beee9284c98ae6193af2c58e8c7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=69aebb264a8d4fbebe1a1e02330fb403]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:43.085222 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e60a8beee9284c98ae6193af2c58e8c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:43.086840 21967 tablet_service.cc:1467] Processing CreateTablet for tablet ab581c1e0054477c849242f57b5066f0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=69aebb264a8d4fbebe1a1e02330fb403]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:43.086891 21969 tablet_service.cc:1467] Processing CreateTablet for tablet 8421b561a3a2402bafee202c129f4c14 (DEFAULT_TABLE table=rebalance_test_table_8 [id=69aebb264a8d4fbebe1a1e02330fb403]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:43.087268 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab581c1e0054477c849242f57b5066f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:43.089438 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8421b561a3a2402bafee202c129f4c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:43.092669 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab581c1e0054477c849242f57b5066f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:43.098088 23541 tablet_bootstrap.cc:492] T 8421b561a3a2402bafee202c129f4c14 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:43.100435 23541 tablet_bootstrap.cc:654] T 8421b561a3a2402bafee202c129f4c14 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:43.112388 23540 tablet_bootstrap.cc:492] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:43.114722 23539 tablet_bootstrap.cc:492] T e60a8beee9284c98ae6193af2c58e8c7 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:43.115782 23541 tablet_bootstrap.cc:492] T 8421b561a3a2402bafee202c129f4c14 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:43.116122 23541 ts_tablet_manager.cc:1367] T 8421b561a3a2402bafee202c129f4c14 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.015s
I20240502 01:53:43.116914 23541 raft_consensus.cc:348] T 8421b561a3a2402bafee202c129f4c14 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:43.117079 23539 tablet_bootstrap.cc:654] T e60a8beee9284c98ae6193af2c58e8c7 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:43.117841 23540 tablet_bootstrap.cc:654] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:43.118741 23541 raft_consensus.cc:374] T 8421b561a3a2402bafee202c129f4c14 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:43.118870 23541 raft_consensus.cc:729] T 8421b561a3a2402bafee202c129f4c14 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:43.118963 23543 tablet_bootstrap.cc:492] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:43.119206 23541 consensus_queue.cc:260] T 8421b561a3a2402bafee202c129f4c14 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:43.119774 23541 ts_tablet_manager.cc:1398] T 8421b561a3a2402bafee202c129f4c14 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:53:43.119788 23539 tablet_bootstrap.cc:492] T e60a8beee9284c98ae6193af2c58e8c7 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:43.119931 23539 ts_tablet_manager.cc:1367] T e60a8beee9284c98ae6193af2c58e8c7 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 01:53:43.120066 23541 tablet_bootstrap.cc:492] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:43.120739 23539 raft_consensus.cc:348] T e60a8beee9284c98ae6193af2c58e8c7 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:43.121073 23539 raft_consensus.cc:374] T e60a8beee9284c98ae6193af2c58e8c7 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:43.121238 23539 raft_consensus.cc:729] T e60a8beee9284c98ae6193af2c58e8c7 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:43.121635 23541 tablet_bootstrap.cc:654] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:43.121546 23539 consensus_queue.cc:260] T e60a8beee9284c98ae6193af2c58e8c7 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:43.123132 23539 ts_tablet_manager.cc:1398] T e60a8beee9284c98ae6193af2c58e8c7 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:53:43.123492 23539 tablet_bootstrap.cc:492] T 8421b561a3a2402bafee202c129f4c14 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:43.124503 23543 tablet_bootstrap.cc:654] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:43.125232 23539 tablet_bootstrap.cc:654] T 8421b561a3a2402bafee202c129f4c14 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:43.128367 23540 tablet_bootstrap.cc:492] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:43.128551 23540 ts_tablet_manager.cc:1367] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240502 01:53:43.129495 23540 raft_consensus.cc:348] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:43.129734 23540 raft_consensus.cc:374] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:43.129834 23540 raft_consensus.cc:729] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:43.130056 23540 consensus_queue.cc:260] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:43.130641 23540 ts_tablet_manager.cc:1398] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:43.130926 23540 tablet_bootstrap.cc:492] T ab581c1e0054477c849242f57b5066f0 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:43.132845 23540 tablet_bootstrap.cc:654] T ab581c1e0054477c849242f57b5066f0 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:43.133036 23539 tablet_bootstrap.cc:492] T 8421b561a3a2402bafee202c129f4c14 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:43.133314 23539 ts_tablet_manager.cc:1367] T 8421b561a3a2402bafee202c129f4c14 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20240502 01:53:43.134212 23539 raft_consensus.cc:348] T 8421b561a3a2402bafee202c129f4c14 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:43.134873 23539 raft_consensus.cc:374] T 8421b561a3a2402bafee202c129f4c14 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:43.135007 23539 raft_consensus.cc:729] T 8421b561a3a2402bafee202c129f4c14 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:43.135442 23541 tablet_bootstrap.cc:492] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:43.135643 23541 ts_tablet_manager.cc:1367] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.005s
I20240502 01:53:43.135447 23539 consensus_queue.cc:260] T 8421b561a3a2402bafee202c129f4c14 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:43.136538 23539 ts_tablet_manager.cc:1398] T 8421b561a3a2402bafee202c129f4c14 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:43.136489 23541 raft_consensus.cc:348] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:43.136669 23541 raft_consensus.cc:374] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:43.136759 23541 raft_consensus.cc:729] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:43.137017 23541 consensus_queue.cc:260] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:43.137667 23541 ts_tablet_manager.cc:1398] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:43.137727 23539 tablet_bootstrap.cc:492] T ab581c1e0054477c849242f57b5066f0 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:43.138062 23540 tablet_bootstrap.cc:492] T ab581c1e0054477c849242f57b5066f0 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:43.138259 23540 ts_tablet_manager.cc:1367] T ab581c1e0054477c849242f57b5066f0 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 01:53:43.139071 23540 raft_consensus.cc:348] T ab581c1e0054477c849242f57b5066f0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:43.139318 23540 raft_consensus.cc:374] T ab581c1e0054477c849242f57b5066f0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:43.139396 23540 raft_consensus.cc:729] T ab581c1e0054477c849242f57b5066f0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:43.139605 23539 tablet_bootstrap.cc:654] T ab581c1e0054477c849242f57b5066f0 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:43.139658 23540 consensus_queue.cc:260] T ab581c1e0054477c849242f57b5066f0 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:43.140329 23540 ts_tablet_manager.cc:1398] T ab581c1e0054477c849242f57b5066f0 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:43.143040 23539 tablet_bootstrap.cc:492] T ab581c1e0054477c849242f57b5066f0 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:43.143241 23539 ts_tablet_manager.cc:1367] T ab581c1e0054477c849242f57b5066f0 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 01:53:43.144106 23539 raft_consensus.cc:348] T ab581c1e0054477c849242f57b5066f0 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:43.144337 23539 raft_consensus.cc:374] T ab581c1e0054477c849242f57b5066f0 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:43.144428 23539 raft_consensus.cc:729] T ab581c1e0054477c849242f57b5066f0 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:43.144680 23539 consensus_queue.cc:260] T ab581c1e0054477c849242f57b5066f0 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:43.145320 23539 ts_tablet_manager.cc:1398] T ab581c1e0054477c849242f57b5066f0 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:43.147356 23543 tablet_bootstrap.cc:492] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:43.147511 23543 ts_tablet_manager.cc:1367] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.008s
I20240502 01:53:43.148283 23543 raft_consensus.cc:348] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:43.148543 23543 raft_consensus.cc:374] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:43.148655 23543 raft_consensus.cc:729] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:43.149029 23543 consensus_queue.cc:260] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:43.149680 23543 ts_tablet_manager.cc:1398] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:43.149971 23543 tablet_bootstrap.cc:492] T e60a8beee9284c98ae6193af2c58e8c7 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:43.151679 23543 tablet_bootstrap.cc:654] T e60a8beee9284c98ae6193af2c58e8c7 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:43.153712 23543 tablet_bootstrap.cc:492] T e60a8beee9284c98ae6193af2c58e8c7 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:43.153908 23543 ts_tablet_manager.cc:1367] T e60a8beee9284c98ae6193af2c58e8c7 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 01:53:43.154733 23543 raft_consensus.cc:348] T e60a8beee9284c98ae6193af2c58e8c7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:43.154909 23543 raft_consensus.cc:374] T e60a8beee9284c98ae6193af2c58e8c7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:43.154965 23543 raft_consensus.cc:729] T e60a8beee9284c98ae6193af2c58e8c7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:43.155187 23543 consensus_queue.cc:260] T e60a8beee9284c98ae6193af2c58e8c7 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:43.155668 23543 ts_tablet_manager.cc:1398] T e60a8beee9284c98ae6193af2c58e8c7 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:43.173877 23366 raft_consensus.cc:482] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:43.174062 23366 raft_consensus.cc:504] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:43.174695 23366 leader_election.cc:288] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:43.175385 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab581c1e0054477c849242f57b5066f0" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:43.175397 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab581c1e0054477c849242f57b5066f0" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:43.175637 22250 raft_consensus.cc:2454] T ab581c1e0054477c849242f57b5066f0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:43.175725 21988 raft_consensus.cc:2454] T ab581c1e0054477c849242f57b5066f0 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:43.176126 21792 leader_election.cc:302] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:43.176425 23366 raft_consensus.cc:2781] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:43.176568 23366 raft_consensus.cc:482] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:43.176705 23366 raft_consensus.cc:3037] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:43.178731 23366 raft_consensus.cc:504] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:43.179286 23366 leader_election.cc:288] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:43.179829 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab581c1e0054477c849242f57b5066f0" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:43.179839 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab581c1e0054477c849242f57b5066f0" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:43.180028 22250 raft_consensus.cc:3037] T ab581c1e0054477c849242f57b5066f0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:43.180059 21989 raft_consensus.cc:3037] T ab581c1e0054477c849242f57b5066f0 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:43.182044 22250 raft_consensus.cc:2454] T ab581c1e0054477c849242f57b5066f0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:43.182063 21989 raft_consensus.cc:2454] T ab581c1e0054477c849242f57b5066f0 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:43.182485 21792 leader_election.cc:302] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:43.182807 23366 raft_consensus.cc:2781] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:43.182960 23366 raft_consensus.cc:686] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:53:43.183256 23366 consensus_queue.cc:237] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:43.186956 21715 catalog_manager.cc:5484] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:53:43.202420 23406 raft_consensus.cc:482] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:43.202628 23406 raft_consensus.cc:504] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:43.203444 23406 leader_election.cc:288] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:43.204061 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60a8beee9284c98ae6193af2c58e8c7" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:43.204095 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60a8beee9284c98ae6193af2c58e8c7" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:43.204341 21856 raft_consensus.cc:2454] T e60a8beee9284c98ae6193af2c58e8c7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:43.204336 21989 raft_consensus.cc:2454] T e60a8beee9284c98ae6193af2c58e8c7 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:43.204818 22053 leader_election.cc:302] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:43.205200 23406 raft_consensus.cc:2781] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:43.205344 23406 raft_consensus.cc:482] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:43.205436 23406 raft_consensus.cc:3037] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:43.207737 23406 raft_consensus.cc:504] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:43.208295 23406 leader_election.cc:288] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:43.208796 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60a8beee9284c98ae6193af2c58e8c7" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:43.208825 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60a8beee9284c98ae6193af2c58e8c7" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:43.209029 21989 raft_consensus.cc:3037] T e60a8beee9284c98ae6193af2c58e8c7 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:43.209087 21856 raft_consensus.cc:3037] T e60a8beee9284c98ae6193af2c58e8c7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:43.211036 21856 raft_consensus.cc:2454] T e60a8beee9284c98ae6193af2c58e8c7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:43.211036 21989 raft_consensus.cc:2454] T e60a8beee9284c98ae6193af2c58e8c7 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:43.211455 22053 leader_election.cc:302] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:43.211773 23406 raft_consensus.cc:2781] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:43.212314 23406 raft_consensus.cc:686] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:53:43.212667 23406 consensus_queue.cc:237] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:43.216037 21715 catalog_manager.cc:5484] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:53:43.216439 23402 raft_consensus.cc:482] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:43.216603 23402 raft_consensus.cc:504] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:43.217412 23402 leader_election.cc:288] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:43.217844 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8421b561a3a2402bafee202c129f4c14" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:43.218000 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8421b561a3a2402bafee202c129f4c14" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:43.218139 22119 raft_consensus.cc:2454] T 8421b561a3a2402bafee202c129f4c14 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:43.218266 21989 raft_consensus.cc:2454] T 8421b561a3a2402bafee202c129f4c14 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:43.218677 22184 leader_election.cc:302] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:43.219130 23402 raft_consensus.cc:2781] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:43.219367 23402 raft_consensus.cc:482] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:43.219489 23402 raft_consensus.cc:3037] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:43.221729 23402 raft_consensus.cc:504] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:43.222326 23402 leader_election.cc:288] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:43.222739 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8421b561a3a2402bafee202c129f4c14" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:43.222754 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8421b561a3a2402bafee202c129f4c14" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:43.222949 22119 raft_consensus.cc:3037] T 8421b561a3a2402bafee202c129f4c14 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:43.222962 21989 raft_consensus.cc:3037] T 8421b561a3a2402bafee202c129f4c14 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:43.225168 22119 raft_consensus.cc:2454] T 8421b561a3a2402bafee202c129f4c14 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:43.225354 21989 raft_consensus.cc:2454] T 8421b561a3a2402bafee202c129f4c14 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:43.225618 22186 leader_election.cc:302] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:43.225920 23402 raft_consensus.cc:2781] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:43.226065 23402 raft_consensus.cc:686] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:53:43.226331 23402 consensus_queue.cc:237] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:43.229707 21715 catalog_manager.cc:5484] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
W20240502 01:53:43.620644 23557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:43.621078 23557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:43.629022 23557 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:53:43.643846 23552 consensus_queue.cc:1035] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:43.650342 23366 consensus_queue.cc:1035] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240502 01:53:43.742103 23406 consensus_queue.cc:1035] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:43.753190 23478 consensus_queue.cc:1035] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:43.759172 23586 consensus_queue.cc:1035] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:43.833860 23478 consensus_queue.cc:1035] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:43.923439 21790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.194:44411, user_credentials={real_user=slave}} blocked reactor thread for 107162us
INSERT report
rows total: 2000
time total: 335.392 ms
time per row: 0.167696 ms
I20240502 01:53:44.128962 21715 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:44.139155 22098 tablet_service.cc:1467] Processing CreateTablet for tablet 4b924192d8da4a8bb7df2e6c5e4cbd4f (DEFAULT_TABLE table=rebalance_test_table_9 [id=714c5d94b5b64cb29dc398713b1727fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.139609 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b924192d8da4a8bb7df2e6c5e4cbd4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.139640 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 166e717fb9624ae58adb6699790daf7a (DEFAULT_TABLE table=rebalance_test_table_9 [id=714c5d94b5b64cb29dc398713b1727fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.140094 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 166e717fb9624ae58adb6699790daf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.144346 23626 tablet_bootstrap.cc:492] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:44.144435 21835 tablet_service.cc:1467] Processing CreateTablet for tablet 4b924192d8da4a8bb7df2e6c5e4cbd4f (DEFAULT_TABLE table=rebalance_test_table_9 [id=714c5d94b5b64cb29dc398713b1727fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.144731 23627 tablet_bootstrap.cc:492] T 166e717fb9624ae58adb6699790daf7a P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:44.144872 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b924192d8da4a8bb7df2e6c5e4cbd4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.145813 21837 tablet_service.cc:1467] Processing CreateTablet for tablet 72f46ec6ed7848438cb9ad6e38053c4d (DEFAULT_TABLE table=rebalance_test_table_9 [id=714c5d94b5b64cb29dc398713b1727fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.146255 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f46ec6ed7848438cb9ad6e38053c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.146373 23626 tablet_bootstrap.cc:654] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.146914 23627 tablet_bootstrap.cc:654] T 166e717fb9624ae58adb6699790daf7a P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.147410 21968 tablet_service.cc:1467] Processing CreateTablet for tablet 72f46ec6ed7848438cb9ad6e38053c4d (DEFAULT_TABLE table=rebalance_test_table_9 [id=714c5d94b5b64cb29dc398713b1727fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.147917 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f46ec6ed7848438cb9ad6e38053c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.154039 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 4b924192d8da4a8bb7df2e6c5e4cbd4f (DEFAULT_TABLE table=rebalance_test_table_9 [id=714c5d94b5b64cb29dc398713b1727fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.154515 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b924192d8da4a8bb7df2e6c5e4cbd4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.154420 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 72f46ec6ed7848438cb9ad6e38053c4d (DEFAULT_TABLE table=rebalance_test_table_9 [id=714c5d94b5b64cb29dc398713b1727fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.154764 22230 tablet_service.cc:1467] Processing CreateTablet for tablet 166e717fb9624ae58adb6699790daf7a (DEFAULT_TABLE table=rebalance_test_table_9 [id=714c5d94b5b64cb29dc398713b1727fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.154896 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f46ec6ed7848438cb9ad6e38053c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.155143 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 166e717fb9624ae58adb6699790daf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.157418 22099 tablet_service.cc:1467] Processing CreateTablet for tablet 166e717fb9624ae58adb6699790daf7a (DEFAULT_TABLE table=rebalance_test_table_9 [id=714c5d94b5b64cb29dc398713b1727fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.157943 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 166e717fb9624ae58adb6699790daf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.164093 23631 tablet_bootstrap.cc:492] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:44.165205 23627 tablet_bootstrap.cc:492] T 166e717fb9624ae58adb6699790daf7a P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:44.165445 23627 ts_tablet_manager.cc:1367] T 166e717fb9624ae58adb6699790daf7a P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20240502 01:53:44.166564 23631 tablet_bootstrap.cc:654] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.168038 23627 raft_consensus.cc:348] T 166e717fb9624ae58adb6699790daf7a P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:44.168480 23627 raft_consensus.cc:374] T 166e717fb9624ae58adb6699790daf7a P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.168640 23627 raft_consensus.cc:729] T 166e717fb9624ae58adb6699790daf7a P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.169076 23627 consensus_queue.cc:260] T 166e717fb9624ae58adb6699790daf7a P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:44.170542 23627 ts_tablet_manager.cc:1398] T 166e717fb9624ae58adb6699790daf7a P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:53:44.171027 23627 tablet_bootstrap.cc:492] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:44.180660 23627 tablet_bootstrap.cc:654] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.181318 23626 tablet_bootstrap.cc:492] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:44.182155 23626 ts_tablet_manager.cc:1367] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.001s
I20240502 01:53:44.183429 23626 raft_consensus.cc:348] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:44.183856 23626 raft_consensus.cc:374] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.184018 23626 raft_consensus.cc:729] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.184468 23626 consensus_queue.cc:260] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:44.185460 23626 ts_tablet_manager.cc:1398] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240502 01:53:44.185914 23626 tablet_bootstrap.cc:492] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:44.188078 23630 tablet_bootstrap.cc:492] T 72f46ec6ed7848438cb9ad6e38053c4d P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:44.188815 23626 tablet_bootstrap.cc:654] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.189535 23627 tablet_bootstrap.cc:492] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:44.189725 23627 ts_tablet_manager.cc:1367] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.001s
I20240502 01:53:44.190542 23630 tablet_bootstrap.cc:654] T 72f46ec6ed7848438cb9ad6e38053c4d P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.190577 23631 tablet_bootstrap.cc:492] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:44.190750 23631 ts_tablet_manager.cc:1367] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.005s
I20240502 01:53:44.190618 23627 raft_consensus.cc:348] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:44.190865 23627 raft_consensus.cc:374] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.190961 23627 raft_consensus.cc:729] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.191591 23631 raft_consensus.cc:348] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:44.192023 23631 raft_consensus.cc:374] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.192154 23631 raft_consensus.cc:729] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.192157 23627 consensus_queue.cc:260] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:44.192443 23631 consensus_queue.cc:260] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:44.193240 23631 ts_tablet_manager.cc:1398] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:53:44.193543 23631 tablet_bootstrap.cc:492] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:44.195228 23631 tablet_bootstrap.cc:654] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.195410 23627 ts_tablet_manager.cc:1398] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 01:53:44.195745 23627 tablet_bootstrap.cc:492] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:44.197540 23627 tablet_bootstrap.cc:654] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.198159 23626 tablet_bootstrap.cc:492] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:44.198410 23626 ts_tablet_manager.cc:1367] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240502 01:53:44.198971 23630 tablet_bootstrap.cc:492] T 72f46ec6ed7848438cb9ad6e38053c4d P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:44.199213 23630 ts_tablet_manager.cc:1367] T 72f46ec6ed7848438cb9ad6e38053c4d P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20240502 01:53:44.199514 23626 raft_consensus.cc:348] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:44.199774 23626 raft_consensus.cc:374] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.199862 23626 raft_consensus.cc:729] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.199992 23627 tablet_bootstrap.cc:492] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:44.200136 23627 ts_tablet_manager.cc:1367] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:44.200160 23626 consensus_queue.cc:260] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:44.200928 23627 raft_consensus.cc:348] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:44.201150 23627 raft_consensus.cc:374] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.201254 23627 raft_consensus.cc:729] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.201632 23626 ts_tablet_manager.cc:1398] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:53:44.201522 23627 consensus_queue.cc:260] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:44.201928 23626 tablet_bootstrap.cc:492] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:44.202180 23627 ts_tablet_manager.cc:1398] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:44.203924 23630 raft_consensus.cc:348] T 72f46ec6ed7848438cb9ad6e38053c4d P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:44.204272 23630 raft_consensus.cc:374] T 72f46ec6ed7848438cb9ad6e38053c4d P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.204432 23630 raft_consensus.cc:729] T 72f46ec6ed7848438cb9ad6e38053c4d P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.204687 23626 tablet_bootstrap.cc:654] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.204913 23630 consensus_queue.cc:260] T 72f46ec6ed7848438cb9ad6e38053c4d P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:44.205804 23630 ts_tablet_manager.cc:1398] T 72f46ec6ed7848438cb9ad6e38053c4d P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:53:44.208973 23626 tablet_bootstrap.cc:492] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:44.209138 23626 ts_tablet_manager.cc:1367] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 01:53:44.209797 23626 raft_consensus.cc:348] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:44.210008 23626 raft_consensus.cc:374] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.210129 23626 raft_consensus.cc:729] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.210378 23626 consensus_queue.cc:260] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:44.210902 23626 ts_tablet_manager.cc:1398] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:44.211720 23631 tablet_bootstrap.cc:492] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:44.211930 23631 ts_tablet_manager.cc:1367] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s
I20240502 01:53:44.212843 23631 raft_consensus.cc:348] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:44.213153 23631 raft_consensus.cc:374] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.213244 23631 raft_consensus.cc:729] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.213517 23631 consensus_queue.cc:260] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:44.214062 23631 ts_tablet_manager.cc:1398] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:44.263870 23406 raft_consensus.cc:482] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:44.264098 23406 raft_consensus.cc:504] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:44.264997 23406 leader_election.cc:288] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:44.265851 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "166e717fb9624ae58adb6699790daf7a" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:44.266117 21856 raft_consensus.cc:2454] T 166e717fb9624ae58adb6699790daf7a P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:44.266865 22053 leader_election.cc:302] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:44.266855 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "166e717fb9624ae58adb6699790daf7a" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:44.267103 22251 raft_consensus.cc:2454] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:44.267180 23406 raft_consensus.cc:2781] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:44.267282 23406 raft_consensus.cc:482] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:44.267366 23406 raft_consensus.cc:3037] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.269802 23406 raft_consensus.cc:504] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:44.270576 23406 leader_election.cc:288] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:44.270985 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "166e717fb9624ae58adb6699790daf7a" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:44.271217 22251 raft_consensus.cc:3037] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.271291 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "166e717fb9624ae58adb6699790daf7a" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:44.271556 21856 raft_consensus.cc:3037] T 166e717fb9624ae58adb6699790daf7a P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.273418 22251 raft_consensus.cc:2454] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:44.273850 22055 leader_election.cc:302] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:44.274163 23406 raft_consensus.cc:2781] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:44.274320 23406 raft_consensus.cc:686] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:53:44.274665 23406 consensus_queue.cc:237] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:44.274876 21856 raft_consensus.cc:2454] T 166e717fb9624ae58adb6699790daf7a P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:44.280086 21715 catalog_manager.cc:5484] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:53:44.353767 23406 raft_consensus.cc:482] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:44.353981 23406 raft_consensus.cc:504] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:44.354609 23406 leader_election.cc:288] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:44.355046 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b924192d8da4a8bb7df2e6c5e4cbd4f" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:44.355126 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b924192d8da4a8bb7df2e6c5e4cbd4f" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:44.355294 21856 raft_consensus.cc:2454] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:44.355366 22251 raft_consensus.cc:2454] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:44.355775 22053 leader_election.cc:302] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:44.356083 23406 raft_consensus.cc:2781] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:44.356174 23406 raft_consensus.cc:482] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:44.356263 23406 raft_consensus.cc:3037] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.358821 23406 raft_consensus.cc:504] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:44.359408 23406 leader_election.cc:288] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:44.359885 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b924192d8da4a8bb7df2e6c5e4cbd4f" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:44.359944 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b924192d8da4a8bb7df2e6c5e4cbd4f" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:44.360090 21856 raft_consensus.cc:3037] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.360193 22251 raft_consensus.cc:3037] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.362435 22251 raft_consensus.cc:2454] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:44.362452 21856 raft_consensus.cc:2454] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:44.362905 22053 leader_election.cc:302] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:44.363185 23406 raft_consensus.cc:2781] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:44.363332 23406 raft_consensus.cc:686] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:53:44.363633 23406 consensus_queue.cc:237] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:44.366714 21715 catalog_manager.cc:5484] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:53:44.380419 23597 raft_consensus.cc:482] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:44.380630 23597 raft_consensus.cc:504] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:44.381358 23597 leader_election.cc:288] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:44.381985 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f46ec6ed7848438cb9ad6e38053c4d" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:44.381989 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f46ec6ed7848438cb9ad6e38053c4d" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:44.382239 22118 raft_consensus.cc:2454] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:44.382256 21988 raft_consensus.cc:2454] T 72f46ec6ed7848438cb9ad6e38053c4d P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:44.382776 21792 leader_election.cc:302] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:44.383073 23597 raft_consensus.cc:2781] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:44.383203 23597 raft_consensus.cc:482] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:44.383292 23597 raft_consensus.cc:3037] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.385367 23597 raft_consensus.cc:504] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:44.385990 23597 leader_election.cc:288] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:44.386462 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f46ec6ed7848438cb9ad6e38053c4d" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:44.386492 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f46ec6ed7848438cb9ad6e38053c4d" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:44.386696 21988 raft_consensus.cc:3037] T 72f46ec6ed7848438cb9ad6e38053c4d P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.386698 22118 raft_consensus.cc:3037] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.388713 22118 raft_consensus.cc:2454] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:44.388751 21988 raft_consensus.cc:2454] T 72f46ec6ed7848438cb9ad6e38053c4d P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:44.389149 21792 leader_election.cc:302] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:44.389477 23597 raft_consensus.cc:2781] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:44.389626 23597 raft_consensus.cc:686] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:53:44.389947 23597 consensus_queue.cc:237] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:44.393507 21715 catalog_manager.cc:5484] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:53:44.658151 23406 consensus_queue.cc:1035] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:44.663749 23406 consensus_queue.cc:1035] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:44.814545 23642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:44.814900 23642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:44.823369 23642 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 01:53:44.888950 23597 consensus_queue.cc:1035] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:44.901369 23366 consensus_queue.cc:1035] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:44.916484 23406 consensus_queue.cc:1035] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:44.925093 23619 consensus_queue.cc:1035] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 237.932 ms
time per row: 0.118966 ms
I20240502 01:53:45.246541 21707 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:45.264691 22231 tablet_service.cc:1467] Processing CreateTablet for tablet dbd6d93ca9224d16a98f11e2e0f1eab0 (DEFAULT_TABLE table=rebalance_test_table_10 [id=17fb52b34dea4f27811e830b6350d802]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:45.264663 21968 tablet_service.cc:1467] Processing CreateTablet for tablet dbd6d93ca9224d16a98f11e2e0f1eab0 (DEFAULT_TABLE table=rebalance_test_table_10 [id=17fb52b34dea4f27811e830b6350d802]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:45.265201 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd6d93ca9224d16a98f11e2e0f1eab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:45.265221 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd6d93ca9224d16a98f11e2e0f1eab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:45.265638 21967 tablet_service.cc:1467] Processing CreateTablet for tablet c2957ed0b72e4be7bdcb475d6c2e4a54 (DEFAULT_TABLE table=rebalance_test_table_10 [id=17fb52b34dea4f27811e830b6350d802]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:45.266093 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2957ed0b72e4be7bdcb475d6c2e4a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:45.267125 21837 tablet_service.cc:1467] Processing CreateTablet for tablet 904443e236824f1db77d9b4eeeeabaf4 (DEFAULT_TABLE table=rebalance_test_table_10 [id=17fb52b34dea4f27811e830b6350d802]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:45.267586 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904443e236824f1db77d9b4eeeeabaf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:45.268422 21835 tablet_service.cc:1467] Processing CreateTablet for tablet dbd6d93ca9224d16a98f11e2e0f1eab0 (DEFAULT_TABLE table=rebalance_test_table_10 [id=17fb52b34dea4f27811e830b6350d802]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:45.268958 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd6d93ca9224d16a98f11e2e0f1eab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:45.269273 21969 tablet_service.cc:1467] Processing CreateTablet for tablet 904443e236824f1db77d9b4eeeeabaf4 (DEFAULT_TABLE table=rebalance_test_table_10 [id=17fb52b34dea4f27811e830b6350d802]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:45.269685 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904443e236824f1db77d9b4eeeeabaf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:45.273684 22099 tablet_service.cc:1467] Processing CreateTablet for tablet c2957ed0b72e4be7bdcb475d6c2e4a54 (DEFAULT_TABLE table=rebalance_test_table_10 [id=17fb52b34dea4f27811e830b6350d802]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:45.274137 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2957ed0b72e4be7bdcb475d6c2e4a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:45.274709 22230 tablet_service.cc:1467] Processing CreateTablet for tablet c2957ed0b72e4be7bdcb475d6c2e4a54 (DEFAULT_TABLE table=rebalance_test_table_10 [id=17fb52b34dea4f27811e830b6350d802]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:45.274760 22229 tablet_service.cc:1467] Processing CreateTablet for tablet 904443e236824f1db77d9b4eeeeabaf4 (DEFAULT_TABLE table=rebalance_test_table_10 [id=17fb52b34dea4f27811e830b6350d802]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:45.275177 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2957ed0b72e4be7bdcb475d6c2e4a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:45.278829 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904443e236824f1db77d9b4eeeeabaf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:45.289474 23709 tablet_bootstrap.cc:492] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:45.291543 23709 tablet_bootstrap.cc:654] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:45.301550 23709 tablet_bootstrap.cc:492] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:45.302330 23709 ts_tablet_manager.cc:1367] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240502 01:53:45.302724 23710 tablet_bootstrap.cc:492] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:45.306942 23711 tablet_bootstrap.cc:492] T 904443e236824f1db77d9b4eeeeabaf4 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:45.307271 23708 tablet_bootstrap.cc:492] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:45.308166 23709 raft_consensus.cc:348] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:45.308523 23709 raft_consensus.cc:374] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:45.308682 23709 raft_consensus.cc:729] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:45.308979 23711 tablet_bootstrap.cc:654] T 904443e236824f1db77d9b4eeeeabaf4 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:45.309063 23709 consensus_queue.cc:260] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:45.309864 23709 ts_tablet_manager.cc:1398] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:45.313380 23710 tablet_bootstrap.cc:654] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:45.318202 23708 tablet_bootstrap.cc:654] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:45.321831 23710 tablet_bootstrap.cc:492] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:45.322134 23710 ts_tablet_manager.cc:1367] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.006s
I20240502 01:53:45.323258 23710 raft_consensus.cc:348] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:45.323601 23710 raft_consensus.cc:374] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:45.323851 23710 raft_consensus.cc:729] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:45.324106 23711 tablet_bootstrap.cc:492] T 904443e236824f1db77d9b4eeeeabaf4 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:45.324329 23711 ts_tablet_manager.cc:1367] T 904443e236824f1db77d9b4eeeeabaf4 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.006s
I20240502 01:53:45.324292 23710 consensus_queue.cc:260] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:45.325178 23711 raft_consensus.cc:348] T 904443e236824f1db77d9b4eeeeabaf4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:45.325453 23711 raft_consensus.cc:374] T 904443e236824f1db77d9b4eeeeabaf4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:45.325587 23711 raft_consensus.cc:729] T 904443e236824f1db77d9b4eeeeabaf4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:45.325937 23711 consensus_queue.cc:260] T 904443e236824f1db77d9b4eeeeabaf4 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:45.327581 23711 ts_tablet_manager.cc:1398] T 904443e236824f1db77d9b4eeeeabaf4 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:45.327925 23711 tablet_bootstrap.cc:492] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:45.329849 23711 tablet_bootstrap.cc:654] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:45.333343 23711 tablet_bootstrap.cc:492] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:45.333379 23710 ts_tablet_manager.cc:1398] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240502 01:53:45.333696 23710 tablet_bootstrap.cc:492] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:45.334619 23711 ts_tablet_manager.cc:1367] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 01:53:45.335448 23710 tablet_bootstrap.cc:654] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:45.335484 23711 raft_consensus.cc:348] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:45.335871 23711 raft_consensus.cc:374] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:45.335968 23711 raft_consensus.cc:729] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:45.336239 23711 consensus_queue.cc:260] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:45.336853 23708 tablet_bootstrap.cc:492] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:45.337149 23708 ts_tablet_manager.cc:1367] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.007s
I20240502 01:53:45.337158 23711 ts_tablet_manager.cc:1398] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:45.337999 23708 raft_consensus.cc:348] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:45.338258 23708 raft_consensus.cc:374] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:45.338402 23708 raft_consensus.cc:729] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:45.338706 23708 consensus_queue.cc:260] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:45.339812 23710 tablet_bootstrap.cc:492] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:45.339977 23710 ts_tablet_manager.cc:1367] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240502 01:53:45.340199 23708 ts_tablet_manager.cc:1398] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:45.340561 23708 tablet_bootstrap.cc:492] T c2957ed0b72e4be7bdcb475d6c2e4a54 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:45.342198 23710 raft_consensus.cc:348] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:45.342428 23710 raft_consensus.cc:374] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:45.342523 23710 raft_consensus.cc:729] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:45.342555 23708 tablet_bootstrap.cc:654] T c2957ed0b72e4be7bdcb475d6c2e4a54 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:45.342828 23710 consensus_queue.cc:260] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:45.343513 23710 ts_tablet_manager.cc:1398] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:45.344133 23710 tablet_bootstrap.cc:492] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:45.346669 23710 tablet_bootstrap.cc:654] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:45.349036 23710 tablet_bootstrap.cc:492] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:45.349200 23710 ts_tablet_manager.cc:1367] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:53:45.349869 23708 tablet_bootstrap.cc:492] T c2957ed0b72e4be7bdcb475d6c2e4a54 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:45.350075 23708 ts_tablet_manager.cc:1367] T c2957ed0b72e4be7bdcb475d6c2e4a54 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20240502 01:53:45.350001 23710 raft_consensus.cc:348] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:45.350222 23710 raft_consensus.cc:374] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:45.350327 23710 raft_consensus.cc:729] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:45.350558 23710 consensus_queue.cc:260] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:45.350891 23708 raft_consensus.cc:348] T c2957ed0b72e4be7bdcb475d6c2e4a54 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:45.351137 23708 raft_consensus.cc:374] T c2957ed0b72e4be7bdcb475d6c2e4a54 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:45.351153 23710 ts_tablet_manager.cc:1398] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:45.351226 23708 raft_consensus.cc:729] T c2957ed0b72e4be7bdcb475d6c2e4a54 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:45.351462 23708 consensus_queue.cc:260] T c2957ed0b72e4be7bdcb475d6c2e4a54 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:45.352005 23708 ts_tablet_manager.cc:1398] T c2957ed0b72e4be7bdcb475d6c2e4a54 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:45.352469 23708 tablet_bootstrap.cc:492] T 904443e236824f1db77d9b4eeeeabaf4 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:45.353924 23708 tablet_bootstrap.cc:654] T 904443e236824f1db77d9b4eeeeabaf4 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:45.358789 23708 tablet_bootstrap.cc:492] T 904443e236824f1db77d9b4eeeeabaf4 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:45.358934 23708 ts_tablet_manager.cc:1367] T 904443e236824f1db77d9b4eeeeabaf4 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240502 01:53:45.359503 23708 raft_consensus.cc:348] T 904443e236824f1db77d9b4eeeeabaf4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:45.359690 23708 raft_consensus.cc:374] T 904443e236824f1db77d9b4eeeeabaf4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:45.359789 23708 raft_consensus.cc:729] T 904443e236824f1db77d9b4eeeeabaf4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:45.359987 23708 consensus_queue.cc:260] T 904443e236824f1db77d9b4eeeeabaf4 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:45.360466 23708 ts_tablet_manager.cc:1398] T 904443e236824f1db77d9b4eeeeabaf4 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:53:45.432108 23704 raft_consensus.cc:482] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:45.432315 23704 raft_consensus.cc:504] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:45.433068 23704 leader_election.cc:288] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:45.433676 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904443e236824f1db77d9b4eeeeabaf4" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:45.433763 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904443e236824f1db77d9b4eeeeabaf4" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:45.433974 21856 raft_consensus.cc:2454] T 904443e236824f1db77d9b4eeeeabaf4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:45.434026 21988 raft_consensus.cc:2454] T 904443e236824f1db77d9b4eeeeabaf4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:45.434464 22184 leader_election.cc:302] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:45.434820 23704 raft_consensus.cc:2781] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:45.434967 23704 raft_consensus.cc:482] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:45.435055 23704 raft_consensus.cc:3037] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:45.437402 23704 raft_consensus.cc:504] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:45.438010 23704 leader_election.cc:288] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:45.438447 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904443e236824f1db77d9b4eeeeabaf4" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:45.438571 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904443e236824f1db77d9b4eeeeabaf4" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:45.438690 21856 raft_consensus.cc:3037] T 904443e236824f1db77d9b4eeeeabaf4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:45.438819 21988 raft_consensus.cc:3037] T 904443e236824f1db77d9b4eeeeabaf4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:45.441735 21856 raft_consensus.cc:2454] T 904443e236824f1db77d9b4eeeeabaf4 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:45.441735 21988 raft_consensus.cc:2454] T 904443e236824f1db77d9b4eeeeabaf4 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:45.442122 22184 leader_election.cc:302] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:45.442442 23704 raft_consensus.cc:2781] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:45.442601 23704 raft_consensus.cc:686] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:53:45.442881 23704 consensus_queue.cc:237] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:45.446223 21707 catalog_manager.cc:5484] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:53:45.499994 23704 raft_consensus.cc:482] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:45.500242 23704 raft_consensus.cc:504] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:45.500950 23704 leader_election.cc:288] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:45.501714 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd6d93ca9224d16a98f11e2e0f1eab0" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:45.501709 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd6d93ca9224d16a98f11e2e0f1eab0" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:45.502002 21988 raft_consensus.cc:2454] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:45.502002 21856 raft_consensus.cc:2454] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:45.502441 22184 leader_election.cc:302] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:45.502792 23704 raft_consensus.cc:2781] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:45.502911 23704 raft_consensus.cc:482] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:45.503024 23704 raft_consensus.cc:3037] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:45.505671 23704 raft_consensus.cc:504] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:45.506421 23704 leader_election.cc:288] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:45.506749 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd6d93ca9224d16a98f11e2e0f1eab0" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:45.506886 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd6d93ca9224d16a98f11e2e0f1eab0" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:45.507072 21856 raft_consensus.cc:3037] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:45.507122 21988 raft_consensus.cc:3037] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:45.509025 21856 raft_consensus.cc:2454] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:45.509033 21988 raft_consensus.cc:2454] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:45.509382 22184 leader_election.cc:302] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:45.509656 23704 raft_consensus.cc:2781] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:45.509846 23704 raft_consensus.cc:686] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:53:45.510082 23704 consensus_queue.cc:237] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:45.510618 23595 raft_consensus.cc:482] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:45.510780 23595 raft_consensus.cc:504] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:45.511610 23595 leader_election.cc:288] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:45.512225 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2957ed0b72e4be7bdcb475d6c2e4a54" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:45.512460 22118 raft_consensus.cc:2454] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:45.512720 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2957ed0b72e4be7bdcb475d6c2e4a54" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:45.512921 22186 leader_election.cc:302] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:45.512979 21988 raft_consensus.cc:2454] T c2957ed0b72e4be7bdcb475d6c2e4a54 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:45.513221 23595 raft_consensus.cc:2781] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:45.513336 23595 raft_consensus.cc:482] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:45.513427 23595 raft_consensus.cc:3037] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:45.513737 21707 catalog_manager.cc:5484] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:53:45.515690 23595 raft_consensus.cc:504] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:45.516947 23595 leader_election.cc:288] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:45.517128 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2957ed0b72e4be7bdcb475d6c2e4a54" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:45.517105 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2957ed0b72e4be7bdcb475d6c2e4a54" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:45.517323 21989 raft_consensus.cc:3037] T c2957ed0b72e4be7bdcb475d6c2e4a54 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:45.517325 22118 raft_consensus.cc:3037] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:45.520206 21989 raft_consensus.cc:2454] T c2957ed0b72e4be7bdcb475d6c2e4a54 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:45.520213 22118 raft_consensus.cc:2454] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:45.520766 22186 leader_election.cc:302] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:45.521097 23595 raft_consensus.cc:2781] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:45.521242 23595 raft_consensus.cc:686] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:53:45.521500 23595 consensus_queue.cc:237] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:45.524511 21707 catalog_manager.cc:5484] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
W20240502 01:53:45.893707 23724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:45.894055 23724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:45.902508 23724 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:53:45.902410 23704 consensus_queue.cc:1035] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:45.909926 23704 consensus_queue.cc:1035] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_10'
I20240502 01:53:45.972061 23704 consensus_queue.cc:1035] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:45.983767 23595 consensus_queue.cc:1035] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:46.004591 23595 consensus_queue.cc:1035] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:46.017881 23704 consensus_queue.cc:1035] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:46.198725 22184 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.193:33785, user_credentials={real_user=slave}} blocked reactor thread for 114566us
W20240502 01:53:46.210916 22053 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.193:33785, user_credentials={real_user=slave}} blocked reactor thread for 104338us
INSERT report
rows total: 2000
time total: 302.895 ms
time per row: 0.151447 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:53:47.364322 21714 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:47.375171 21968 tablet_service.cc:1467] Processing CreateTablet for tablet 8a0ae0d8e8784db9988b3469d318f7fb (DEFAULT_TABLE table=rebalance_test_table_11 [id=e67a4afb37114119a9d514153a1563bb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:47.375473 21837 tablet_service.cc:1467] Processing CreateTablet for tablet 8a0ae0d8e8784db9988b3469d318f7fb (DEFAULT_TABLE table=rebalance_test_table_11 [id=e67a4afb37114119a9d514153a1563bb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:47.375598 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a0ae0d8e8784db9988b3469d318f7fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.375888 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a0ae0d8e8784db9988b3469d318f7fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.380519 23790 tablet_bootstrap.cc:492] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:47.381357 22230 tablet_service.cc:1467] Processing CreateTablet for tablet 3964f5f080f145a5923dac2f076637c8 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e67a4afb37114119a9d514153a1563bb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:47.381392 22099 tablet_service.cc:1467] Processing CreateTablet for tablet 8a0ae0d8e8784db9988b3469d318f7fb (DEFAULT_TABLE table=rebalance_test_table_11 [id=e67a4afb37114119a9d514153a1563bb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:47.381528 22098 tablet_service.cc:1467] Processing CreateTablet for tablet 3964f5f080f145a5923dac2f076637c8 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e67a4afb37114119a9d514153a1563bb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:47.381837 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3964f5f080f145a5923dac2f076637c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.381856 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a0ae0d8e8784db9988b3469d318f7fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.382148 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3964f5f080f145a5923dac2f076637c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.382534 23790 tablet_bootstrap.cc:654] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:47.387485 21969 tablet_service.cc:1467] Processing CreateTablet for tablet 3964f5f080f145a5923dac2f076637c8 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e67a4afb37114119a9d514153a1563bb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:47.387940 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3964f5f080f145a5923dac2f076637c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.388118 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 9bceab60366b4ade98f86072d391e7f4 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e67a4afb37114119a9d514153a1563bb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:47.388480 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bceab60366b4ade98f86072d391e7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.396390 23791 tablet_bootstrap.cc:492] T 8a0ae0d8e8784db9988b3469d318f7fb P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:47.397843 21835 tablet_service.cc:1467] Processing CreateTablet for tablet 9bceab60366b4ade98f86072d391e7f4 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e67a4afb37114119a9d514153a1563bb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:47.398479 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bceab60366b4ade98f86072d391e7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.399068 22229 tablet_service.cc:1467] Processing CreateTablet for tablet 9bceab60366b4ade98f86072d391e7f4 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e67a4afb37114119a9d514153a1563bb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:47.399627 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bceab60366b4ade98f86072d391e7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.398473 23791 tablet_bootstrap.cc:654] T 8a0ae0d8e8784db9988b3469d318f7fb P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:47.405635 23794 tablet_bootstrap.cc:492] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:47.408200 23794 tablet_bootstrap.cc:654] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:47.410742 23793 tablet_bootstrap.cc:492] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:47.410871 23790 tablet_bootstrap.cc:492] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:47.411058 23790 ts_tablet_manager.cc:1367] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.017s
I20240502 01:53:47.411288 23791 tablet_bootstrap.cc:492] T 8a0ae0d8e8784db9988b3469d318f7fb P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:47.411463 23791 ts_tablet_manager.cc:1367] T 8a0ae0d8e8784db9988b3469d318f7fb P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
I20240502 01:53:47.411792 23790 raft_consensus.cc:348] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:47.411972 23790 raft_consensus.cc:374] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:47.412083 23790 raft_consensus.cc:729] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:47.412299 23790 consensus_queue.cc:260] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:47.412276 23791 raft_consensus.cc:348] T 8a0ae0d8e8784db9988b3469d318f7fb P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:47.412484 23791 raft_consensus.cc:374] T 8a0ae0d8e8784db9988b3469d318f7fb P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:47.412629 23791 raft_consensus.cc:729] T 8a0ae0d8e8784db9988b3469d318f7fb P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:47.412662 23793 tablet_bootstrap.cc:654] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:47.413024 23791 consensus_queue.cc:260] T 8a0ae0d8e8784db9988b3469d318f7fb P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:47.414322 23790 ts_tablet_manager.cc:1398] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:53:47.414875 23791 ts_tablet_manager.cc:1398] T 8a0ae0d8e8784db9988b3469d318f7fb P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:53:47.415222 23791 tablet_bootstrap.cc:492] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:47.415485 23790 tablet_bootstrap.cc:492] T 3964f5f080f145a5923dac2f076637c8 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:47.417340 23790 tablet_bootstrap.cc:654] T 3964f5f080f145a5923dac2f076637c8 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:47.417359 23791 tablet_bootstrap.cc:654] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:47.419596 23794 tablet_bootstrap.cc:492] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:47.419790 23794 ts_tablet_manager.cc:1367] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240502 01:53:47.422852 23794 raft_consensus.cc:348] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:47.423154 23794 raft_consensus.cc:374] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:47.423259 23794 raft_consensus.cc:729] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:47.425383 23790 tablet_bootstrap.cc:492] T 3964f5f080f145a5923dac2f076637c8 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:47.423529 23794 consensus_queue.cc:260] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:47.425570 23790 ts_tablet_manager.cc:1367] T 3964f5f080f145a5923dac2f076637c8 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240502 01:53:47.426081 23794 ts_tablet_manager.cc:1398] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 01:53:47.426625 23794 tablet_bootstrap.cc:492] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:47.427256 23790 raft_consensus.cc:348] T 3964f5f080f145a5923dac2f076637c8 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:47.427695 23790 raft_consensus.cc:374] T 3964f5f080f145a5923dac2f076637c8 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:47.427783 23790 raft_consensus.cc:729] T 3964f5f080f145a5923dac2f076637c8 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:47.428022 23790 consensus_queue.cc:260] T 3964f5f080f145a5923dac2f076637c8 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:47.428539 23790 ts_tablet_manager.cc:1398] T 3964f5f080f145a5923dac2f076637c8 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:47.428836 23794 tablet_bootstrap.cc:654] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:47.429421 23791 tablet_bootstrap.cc:492] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:47.429603 23791 ts_tablet_manager.cc:1367] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20240502 01:53:47.430398 23791 raft_consensus.cc:348] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:47.430646 23791 raft_consensus.cc:374] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:47.430744 23791 raft_consensus.cc:729] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:47.430859 23794 tablet_bootstrap.cc:492] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:47.431033 23794 ts_tablet_manager.cc:1367] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 01:53:47.431371 23791 consensus_queue.cc:260] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:47.431826 23793 tablet_bootstrap.cc:492] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:47.432008 23793 ts_tablet_manager.cc:1367] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20240502 01:53:47.432906 23791 ts_tablet_manager.cc:1398] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:47.432823 23793 raft_consensus.cc:348] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:47.433072 23793 raft_consensus.cc:374] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:47.433153 23793 raft_consensus.cc:729] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:47.433351 23793 consensus_queue.cc:260] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:47.433801 23793 ts_tablet_manager.cc:1398] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:47.434092 23793 tablet_bootstrap.cc:492] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:47.435705 23793 tablet_bootstrap.cc:654] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:47.437476 23794 raft_consensus.cc:348] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:47.437819 23794 raft_consensus.cc:374] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:47.437948 23794 raft_consensus.cc:729] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:47.438241 23794 consensus_queue.cc:260] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:47.438843 23794 ts_tablet_manager.cc:1398] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240502 01:53:47.439960 23793 tablet_bootstrap.cc:492] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:47.440187 23793 ts_tablet_manager.cc:1367] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 01:53:47.441725 23793 raft_consensus.cc:348] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:47.441967 23793 raft_consensus.cc:374] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:47.442066 23793 raft_consensus.cc:729] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:47.442328 23793 consensus_queue.cc:260] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:47.442929 23793 ts_tablet_manager.cc:1398] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:47.443352 23793 tablet_bootstrap.cc:492] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:47.444833 23793 tablet_bootstrap.cc:654] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:47.446611 23793 tablet_bootstrap.cc:492] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:47.446794 23793 ts_tablet_manager.cc:1367] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:47.447471 23793 raft_consensus.cc:348] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:47.447665 23793 raft_consensus.cc:374] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:47.447775 23793 raft_consensus.cc:729] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:47.448012 23793 consensus_queue.cc:260] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:47.448496 23793 ts_tablet_manager.cc:1398] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:47.471345 23784 raft_consensus.cc:482] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:47.471535 23784 raft_consensus.cc:504] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:47.472141 23784 leader_election.cc:288] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:47.472672 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0ae0d8e8784db9988b3469d318f7fb" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:47.472729 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0ae0d8e8784db9988b3469d318f7fb" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:47.472920 22119 raft_consensus.cc:2454] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:47.473014 21856 raft_consensus.cc:2454] T 8a0ae0d8e8784db9988b3469d318f7fb P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:47.473474 21924 leader_election.cc:302] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:47.473776 23784 raft_consensus.cc:2781] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:47.473877 23784 raft_consensus.cc:482] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:47.473968 23784 raft_consensus.cc:3037] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:47.475899 23784 raft_consensus.cc:504] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:47.476509 23784 leader_election.cc:288] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:47.477023 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0ae0d8e8784db9988b3469d318f7fb" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:47.477025 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0ae0d8e8784db9988b3469d318f7fb" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:47.477218 21856 raft_consensus.cc:3037] T 8a0ae0d8e8784db9988b3469d318f7fb P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:47.477224 22119 raft_consensus.cc:3037] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:47.479569 22119 raft_consensus.cc:2454] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:47.479575 21856 raft_consensus.cc:2454] T 8a0ae0d8e8784db9988b3469d318f7fb P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:47.479980 21922 leader_election.cc:302] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:53:47.480266 23784 raft_consensus.cc:2781] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:47.480427 23784 raft_consensus.cc:686] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:53:47.480729 23784 consensus_queue.cc:237] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:47.483767 21709 catalog_manager.cc:5484] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:53:47.488284 23803 raft_consensus.cc:482] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:47.488451 23803 raft_consensus.cc:504] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:47.489125 23803 leader_election.cc:288] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:47.489595 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bceab60366b4ade98f86072d391e7f4" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:47.489681 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bceab60366b4ade98f86072d391e7f4" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:47.489815 21856 raft_consensus.cc:2454] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:47.489935 22250 raft_consensus.cc:2454] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:47.490259 22053 leader_election.cc:302] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:47.490559 23803 raft_consensus.cc:2781] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:47.490687 23803 raft_consensus.cc:482] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:47.490777 23803 raft_consensus.cc:3037] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:47.492681 23803 raft_consensus.cc:504] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:47.493238 23803 leader_election.cc:288] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:47.493664 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bceab60366b4ade98f86072d391e7f4" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:47.493783 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bceab60366b4ade98f86072d391e7f4" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:47.493921 21856 raft_consensus.cc:3037] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:47.494006 22250 raft_consensus.cc:3037] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:47.496874 21856 raft_consensus.cc:2454] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:47.496928 22250 raft_consensus.cc:2454] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:47.497298 22055 leader_election.cc:302] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:47.497614 23803 raft_consensus.cc:2781] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:47.497743 23803 raft_consensus.cc:686] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:53:47.498020 23803 consensus_queue.cc:237] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:47.501298 21709 catalog_manager.cc:5484] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:53:47.571856 23769 raft_consensus.cc:482] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:47.572033 23769 raft_consensus.cc:504] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:47.572701 23769 leader_election.cc:288] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:47.573137 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3964f5f080f145a5923dac2f076637c8" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:47.573284 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3964f5f080f145a5923dac2f076637c8" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:47.573366 22119 raft_consensus.cc:2454] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:47.573530 21988 raft_consensus.cc:2454] T 3964f5f080f145a5923dac2f076637c8 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:47.573818 22186 leader_election.cc:302] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:47.574093 23769 raft_consensus.cc:2781] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:47.574206 23769 raft_consensus.cc:482] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:47.574298 23769 raft_consensus.cc:3037] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:47.576331 23769 raft_consensus.cc:504] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:47.577071 23769 leader_election.cc:288] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:47.577425 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3964f5f080f145a5923dac2f076637c8" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:47.577522 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3964f5f080f145a5923dac2f076637c8" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:47.577757 21988 raft_consensus.cc:3037] T 3964f5f080f145a5923dac2f076637c8 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:47.577790 22119 raft_consensus.cc:3037] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:47.579779 21988 raft_consensus.cc:2454] T 3964f5f080f145a5923dac2f076637c8 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:47.579928 22119 raft_consensus.cc:2454] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:47.580164 22184 leader_election.cc:302] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:47.580449 23769 raft_consensus.cc:2781] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:47.580600 23769 raft_consensus.cc:686] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:53:47.580914 23769 consensus_queue.cc:237] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:47.584184 21709 catalog_manager.cc:5484] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:53:47.929400 23803 consensus_queue.cc:1035] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:47.937074 23803 consensus_queue.cc:1035] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:47.959991 23807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:47.960345 23807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:47.968140 23807 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:53:47.991462 23784 consensus_queue.cc:1035] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:47.997884 23784 consensus_queue.cc:1035] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_11'
I20240502 01:53:48.108292 23769 consensus_queue.cc:1035] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:48.151105 23769 consensus_queue.cc:1035] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:48.228453 21792 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.195:38879, user_credentials={real_user=slave}} blocked reactor thread for 46893.5us
INSERT report
rows total: 2000
time total: 381.247 ms
time per row: 0.190623 ms
I20240502 01:53:48.513762 21712 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:48.525085 21969 tablet_service.cc:1467] Processing CreateTablet for tablet 913d652575644bd4a8d25ba085e1715e (DEFAULT_TABLE table=rebalance_test_table_12 [id=906b29d0bc414f019de158cfa649177d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:48.525089 22098 tablet_service.cc:1467] Processing CreateTablet for tablet 913d652575644bd4a8d25ba085e1715e (DEFAULT_TABLE table=rebalance_test_table_12 [id=906b29d0bc414f019de158cfa649177d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:48.525269 22229 tablet_service.cc:1467] Processing CreateTablet for tablet 26138bfc68834edaa4dcda8edcb5d69c (DEFAULT_TABLE table=rebalance_test_table_12 [id=906b29d0bc414f019de158cfa649177d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:48.525620 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913d652575644bd4a8d25ba085e1715e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.525632 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913d652575644bd4a8d25ba085e1715e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.525683 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26138bfc68834edaa4dcda8edcb5d69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.526114 21835 tablet_service.cc:1467] Processing CreateTablet for tablet 913d652575644bd4a8d25ba085e1715e (DEFAULT_TABLE table=rebalance_test_table_12 [id=906b29d0bc414f019de158cfa649177d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:48.526605 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913d652575644bd4a8d25ba085e1715e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.533769 22230 tablet_service.cc:1467] Processing CreateTablet for tablet c3acd37f59844c0783643e9aef41ea9c (DEFAULT_TABLE table=rebalance_test_table_12 [id=906b29d0bc414f019de158cfa649177d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:48.534101 22100 tablet_service.cc:1467] Processing CreateTablet for tablet c3acd37f59844c0783643e9aef41ea9c (DEFAULT_TABLE table=rebalance_test_table_12 [id=906b29d0bc414f019de158cfa649177d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:48.534330 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3acd37f59844c0783643e9aef41ea9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.534577 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3acd37f59844c0783643e9aef41ea9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.534404 21968 tablet_service.cc:1467] Processing CreateTablet for tablet 26138bfc68834edaa4dcda8edcb5d69c (DEFAULT_TABLE table=rebalance_test_table_12 [id=906b29d0bc414f019de158cfa649177d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:48.534970 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26138bfc68834edaa4dcda8edcb5d69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.536517 23880 tablet_bootstrap.cc:492] T 26138bfc68834edaa4dcda8edcb5d69c P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:48.537608 21967 tablet_service.cc:1467] Processing CreateTablet for tablet c3acd37f59844c0783643e9aef41ea9c (DEFAULT_TABLE table=rebalance_test_table_12 [id=906b29d0bc414f019de158cfa649177d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:48.538110 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3acd37f59844c0783643e9aef41ea9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.539800 23881 tablet_bootstrap.cc:492] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:48.539790 21837 tablet_service.cc:1467] Processing CreateTablet for tablet 26138bfc68834edaa4dcda8edcb5d69c (DEFAULT_TABLE table=rebalance_test_table_12 [id=906b29d0bc414f019de158cfa649177d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:48.540249 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26138bfc68834edaa4dcda8edcb5d69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.541365 23881 tablet_bootstrap.cc:654] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.542317 23880 tablet_bootstrap.cc:654] T 26138bfc68834edaa4dcda8edcb5d69c P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.544238 23881 tablet_bootstrap.cc:492] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:48.544365 23881 ts_tablet_manager.cc:1367] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:53:48.545421 23881 raft_consensus.cc:348] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:48.545688 23881 raft_consensus.cc:374] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.545799 23881 raft_consensus.cc:729] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.546041 23881 consensus_queue.cc:260] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:48.546563 23881 ts_tablet_manager.cc:1398] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:48.546875 23881 tablet_bootstrap.cc:492] T c3acd37f59844c0783643e9aef41ea9c P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:48.549008 23880 tablet_bootstrap.cc:492] T 26138bfc68834edaa4dcda8edcb5d69c P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:48.549201 23880 ts_tablet_manager.cc:1367] T 26138bfc68834edaa4dcda8edcb5d69c P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240502 01:53:48.549366 23881 tablet_bootstrap.cc:654] T c3acd37f59844c0783643e9aef41ea9c P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.550109 23880 raft_consensus.cc:348] T 26138bfc68834edaa4dcda8edcb5d69c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:48.550410 23880 raft_consensus.cc:374] T 26138bfc68834edaa4dcda8edcb5d69c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.550521 23880 raft_consensus.cc:729] T 26138bfc68834edaa4dcda8edcb5d69c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.550843 23880 consensus_queue.cc:260] T 26138bfc68834edaa4dcda8edcb5d69c P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:48.551693 23880 ts_tablet_manager.cc:1398] T 26138bfc68834edaa4dcda8edcb5d69c P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:48.559782 23881 tablet_bootstrap.cc:492] T c3acd37f59844c0783643e9aef41ea9c P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:48.560299 23881 ts_tablet_manager.cc:1367] T c3acd37f59844c0783643e9aef41ea9c P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20240502 01:53:48.561225 23881 raft_consensus.cc:348] T c3acd37f59844c0783643e9aef41ea9c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:48.561496 23881 raft_consensus.cc:374] T c3acd37f59844c0783643e9aef41ea9c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.561591 23881 raft_consensus.cc:729] T c3acd37f59844c0783643e9aef41ea9c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.562151 23881 consensus_queue.cc:260] T c3acd37f59844c0783643e9aef41ea9c P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:48.562795 23881 ts_tablet_manager.cc:1398] T c3acd37f59844c0783643e9aef41ea9c P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:48.569005 23880 tablet_bootstrap.cc:492] T c3acd37f59844c0783643e9aef41ea9c P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:48.570863 23880 tablet_bootstrap.cc:654] T c3acd37f59844c0783643e9aef41ea9c P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.572062 23886 tablet_bootstrap.cc:492] T 913d652575644bd4a8d25ba085e1715e P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:48.573642 23884 tablet_bootstrap.cc:492] T 913d652575644bd4a8d25ba085e1715e P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:48.574297 23886 tablet_bootstrap.cc:654] T 913d652575644bd4a8d25ba085e1715e P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.575855 23884 tablet_bootstrap.cc:654] T 913d652575644bd4a8d25ba085e1715e P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.576326 23880 tablet_bootstrap.cc:492] T c3acd37f59844c0783643e9aef41ea9c P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:48.576522 23880 ts_tablet_manager.cc:1367] T c3acd37f59844c0783643e9aef41ea9c P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 01:53:48.576756 23886 tablet_bootstrap.cc:492] T 913d652575644bd4a8d25ba085e1715e P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:48.576929 23886 ts_tablet_manager.cc:1367] T 913d652575644bd4a8d25ba085e1715e P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:53:48.577515 23880 raft_consensus.cc:348] T c3acd37f59844c0783643e9aef41ea9c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:48.577759 23880 raft_consensus.cc:374] T c3acd37f59844c0783643e9aef41ea9c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.577838 23880 raft_consensus.cc:729] T c3acd37f59844c0783643e9aef41ea9c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.577951 23886 raft_consensus.cc:348] T 913d652575644bd4a8d25ba085e1715e P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:48.578104 23880 consensus_queue.cc:260] T c3acd37f59844c0783643e9aef41ea9c P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:48.578245 23886 raft_consensus.cc:374] T 913d652575644bd4a8d25ba085e1715e P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.578374 23886 raft_consensus.cc:729] T 913d652575644bd4a8d25ba085e1715e P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.578686 23880 ts_tablet_manager.cc:1398] T c3acd37f59844c0783643e9aef41ea9c P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:48.579748 23886 consensus_queue.cc:260] T 913d652575644bd4a8d25ba085e1715e P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:48.580718 23884 tablet_bootstrap.cc:492] T 913d652575644bd4a8d25ba085e1715e P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:48.580971 23884 ts_tablet_manager.cc:1367] T 913d652575644bd4a8d25ba085e1715e P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 01:53:48.582419 23884 raft_consensus.cc:348] T 913d652575644bd4a8d25ba085e1715e P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:48.582715 23884 raft_consensus.cc:374] T 913d652575644bd4a8d25ba085e1715e P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.583042 23884 raft_consensus.cc:729] T 913d652575644bd4a8d25ba085e1715e P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.583384 23884 consensus_queue.cc:260] T 913d652575644bd4a8d25ba085e1715e P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:48.584234 23884 ts_tablet_manager.cc:1398] T 913d652575644bd4a8d25ba085e1715e P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:48.584636 23884 tablet_bootstrap.cc:492] T 26138bfc68834edaa4dcda8edcb5d69c P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:48.586777 23884 tablet_bootstrap.cc:654] T 26138bfc68834edaa4dcda8edcb5d69c P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.589435 23884 tablet_bootstrap.cc:492] T 26138bfc68834edaa4dcda8edcb5d69c P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:48.589632 23884 ts_tablet_manager.cc:1367] T 26138bfc68834edaa4dcda8edcb5d69c P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:53:48.590466 23884 raft_consensus.cc:348] T 26138bfc68834edaa4dcda8edcb5d69c P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:48.590804 23884 raft_consensus.cc:374] T 26138bfc68834edaa4dcda8edcb5d69c P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.590939 23884 raft_consensus.cc:729] T 26138bfc68834edaa4dcda8edcb5d69c P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.591125 23886 ts_tablet_manager.cc:1398] T 913d652575644bd4a8d25ba085e1715e P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 01:53:48.591271 23884 consensus_queue.cc:260] T 26138bfc68834edaa4dcda8edcb5d69c P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:48.591948 23884 ts_tablet_manager.cc:1398] T 26138bfc68834edaa4dcda8edcb5d69c P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:48.592602 23886 tablet_bootstrap.cc:492] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:48.594614 23886 tablet_bootstrap.cc:654] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.598273 23886 tablet_bootstrap.cc:492] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:48.598520 23886 ts_tablet_manager.cc:1367] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240502 01:53:48.599293 23886 raft_consensus.cc:348] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:48.599499 23886 raft_consensus.cc:374] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.599575 23886 raft_consensus.cc:729] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.599851 23886 consensus_queue.cc:260] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:48.600420 23886 ts_tablet_manager.cc:1398] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:48.600765 23886 tablet_bootstrap.cc:492] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:48.602192 23886 tablet_bootstrap.cc:654] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.604295 23886 tablet_bootstrap.cc:492] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:48.604461 23886 ts_tablet_manager.cc:1367] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:48.605234 23886 raft_consensus.cc:348] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:48.605438 23886 raft_consensus.cc:374] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.605532 23886 raft_consensus.cc:729] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.605890 23886 consensus_queue.cc:260] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:48.606509 23886 ts_tablet_manager.cc:1398] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:48.660301 23784 raft_consensus.cc:482] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:48.660522 23784 raft_consensus.cc:504] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:48.661293 23784 leader_election.cc:288] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:48.661819 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26138bfc68834edaa4dcda8edcb5d69c" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:48.661875 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26138bfc68834edaa4dcda8edcb5d69c" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:48.662076 21857 raft_consensus.cc:2454] T 26138bfc68834edaa4dcda8edcb5d69c P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:48.662123 22250 raft_consensus.cc:2454] T 26138bfc68834edaa4dcda8edcb5d69c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:48.662595 21922 leader_election.cc:302] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:53:48.662901 23784 raft_consensus.cc:2781] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:48.663010 23784 raft_consensus.cc:482] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:48.663168 23784 raft_consensus.cc:3037] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.665340 23784 raft_consensus.cc:504] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:48.665988 23784 leader_election.cc:288] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:48.666476 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26138bfc68834edaa4dcda8edcb5d69c" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:48.666584 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26138bfc68834edaa4dcda8edcb5d69c" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:48.666721 22250 raft_consensus.cc:3037] T 26138bfc68834edaa4dcda8edcb5d69c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.666795 21857 raft_consensus.cc:3037] T 26138bfc68834edaa4dcda8edcb5d69c P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.669735 21857 raft_consensus.cc:2454] T 26138bfc68834edaa4dcda8edcb5d69c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:48.669726 22250 raft_consensus.cc:2454] T 26138bfc68834edaa4dcda8edcb5d69c P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:48.670217 21924 leader_election.cc:302] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:48.670509 23784 raft_consensus.cc:2781] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:48.670703 23784 raft_consensus.cc:686] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:53:48.671012 23784 consensus_queue.cc:237] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:48.674422 21714 catalog_manager.cc:5484] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:53:48.703801 23784 raft_consensus.cc:482] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:48.704008 23784 raft_consensus.cc:504] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:48.704726 23784 leader_election.cc:288] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:48.705233 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3acd37f59844c0783643e9aef41ea9c" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:48.705312 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3acd37f59844c0783643e9aef41ea9c" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:48.705524 22119 raft_consensus.cc:2454] T c3acd37f59844c0783643e9aef41ea9c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:48.705528 22250 raft_consensus.cc:2454] T c3acd37f59844c0783643e9aef41ea9c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:48.705952 21924 leader_election.cc:302] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:48.706252 23784 raft_consensus.cc:2781] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:48.706379 23784 raft_consensus.cc:482] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:48.706475 23784 raft_consensus.cc:3037] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.708954 23784 raft_consensus.cc:504] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:48.709619 23784 leader_election.cc:288] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:48.710032 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3acd37f59844c0783643e9aef41ea9c" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:48.710080 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3acd37f59844c0783643e9aef41ea9c" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:48.710278 22119 raft_consensus.cc:3037] T c3acd37f59844c0783643e9aef41ea9c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.710292 22250 raft_consensus.cc:3037] T c3acd37f59844c0783643e9aef41ea9c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.713151 22250 raft_consensus.cc:2454] T c3acd37f59844c0783643e9aef41ea9c P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:48.713176 22119 raft_consensus.cc:2454] T c3acd37f59844c0783643e9aef41ea9c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:48.713618 21924 leader_election.cc:302] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:48.714246 23784 raft_consensus.cc:2781] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:48.714413 23784 raft_consensus.cc:686] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:53:48.714758 23784 consensus_queue.cc:237] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:48.718650 21714 catalog_manager.cc:5484] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:53:48.764559 23874 raft_consensus.cc:482] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:48.764737 23874 raft_consensus.cc:504] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:48.765594 23874 leader_election.cc:288] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:48.765987 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913d652575644bd4a8d25ba085e1715e" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:48.766208 21857 raft_consensus.cc:2454] T 913d652575644bd4a8d25ba085e1715e P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:48.766142 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913d652575644bd4a8d25ba085e1715e" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:48.766386 21988 raft_consensus.cc:2454] T 913d652575644bd4a8d25ba085e1715e P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:48.766711 22053 leader_election.cc:302] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:48.766993 23874 raft_consensus.cc:2781] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:48.767131 23874 raft_consensus.cc:482] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:48.767226 23874 raft_consensus.cc:3037] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.769304 23874 raft_consensus.cc:504] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:48.769882 23874 leader_election.cc:288] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:48.770359 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913d652575644bd4a8d25ba085e1715e" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:48.770386 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913d652575644bd4a8d25ba085e1715e" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:48.770584 21857 raft_consensus.cc:3037] T 913d652575644bd4a8d25ba085e1715e P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.770615 21988 raft_consensus.cc:3037] T 913d652575644bd4a8d25ba085e1715e P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.772521 21988 raft_consensus.cc:2454] T 913d652575644bd4a8d25ba085e1715e P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:48.772588 21857 raft_consensus.cc:2454] T 913d652575644bd4a8d25ba085e1715e P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:48.772953 22053 leader_election.cc:302] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:48.773276 23874 raft_consensus.cc:2781] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:48.773432 23874 raft_consensus.cc:686] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:53:48.773728 23874 consensus_queue.cc:237] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:48.776852 21714 catalog_manager.cc:5484] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
W20240502 01:53:49.161324 23896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:49.161691 23896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:49.169916 23896 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240502 01:53:49.239434 23784 consensus_queue.cc:1035] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:49.244072 23874 consensus_queue.cc:1035] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:49.254518 23803 consensus_queue.cc:1035] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:49.271366 23784 consensus_queue.cc:1035] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:49.287792 23927 consensus_queue.cc:1035] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:49.302309 23927 consensus_queue.cc:1035] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 306.487 ms
time per row: 0.153244 ms
I20240502 01:53:49.630978 21714 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:49.640926 21835 tablet_service.cc:1467] Processing CreateTablet for tablet d81f9cab671a4258aee1dd4ea4a5ee6b (DEFAULT_TABLE table=rebalance_test_table_13 [id=b462b431b51b4d45baf0ff5a495aa90a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:49.641463 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81f9cab671a4258aee1dd4ea4a5ee6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.641667 22100 tablet_service.cc:1467] Processing CreateTablet for tablet d81f9cab671a4258aee1dd4ea4a5ee6b (DEFAULT_TABLE table=rebalance_test_table_13 [id=b462b431b51b4d45baf0ff5a495aa90a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:49.641633 22229 tablet_service.cc:1467] Processing CreateTablet for tablet 3be66f4b90824a21a5f41915c9041970 (DEFAULT_TABLE table=rebalance_test_table_13 [id=b462b431b51b4d45baf0ff5a495aa90a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:49.642100 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81f9cab671a4258aee1dd4ea4a5ee6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.642102 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be66f4b90824a21a5f41915c9041970. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.642532 21836 tablet_service.cc:1467] Processing CreateTablet for tablet fedbec8860904a2cb5294037b7a4bcad (DEFAULT_TABLE table=rebalance_test_table_13 [id=b462b431b51b4d45baf0ff5a495aa90a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:49.642992 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fedbec8860904a2cb5294037b7a4bcad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.644702 21837 tablet_service.cc:1467] Processing CreateTablet for tablet 3be66f4b90824a21a5f41915c9041970 (DEFAULT_TABLE table=rebalance_test_table_13 [id=b462b431b51b4d45baf0ff5a495aa90a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:49.645116 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be66f4b90824a21a5f41915c9041970. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.650068 22099 tablet_service.cc:1467] Processing CreateTablet for tablet fedbec8860904a2cb5294037b7a4bcad (DEFAULT_TABLE table=rebalance_test_table_13 [id=b462b431b51b4d45baf0ff5a495aa90a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:49.650146 22098 tablet_service.cc:1467] Processing CreateTablet for tablet 3be66f4b90824a21a5f41915c9041970 (DEFAULT_TABLE table=rebalance_test_table_13 [id=b462b431b51b4d45baf0ff5a495aa90a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:49.650525 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fedbec8860904a2cb5294037b7a4bcad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.643821 22230 tablet_service.cc:1467] Processing CreateTablet for tablet d81f9cab671a4258aee1dd4ea4a5ee6b (DEFAULT_TABLE table=rebalance_test_table_13 [id=b462b431b51b4d45baf0ff5a495aa90a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:49.651608 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81f9cab671a4258aee1dd4ea4a5ee6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.653257 21969 tablet_service.cc:1467] Processing CreateTablet for tablet fedbec8860904a2cb5294037b7a4bcad (DEFAULT_TABLE table=rebalance_test_table_13 [id=b462b431b51b4d45baf0ff5a495aa90a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:49.653692 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fedbec8860904a2cb5294037b7a4bcad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.657061 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be66f4b90824a21a5f41915c9041970. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.670878 23954 tablet_bootstrap.cc:492] T d81f9cab671a4258aee1dd4ea4a5ee6b P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:49.674021 23954 tablet_bootstrap.cc:654] T d81f9cab671a4258aee1dd4ea4a5ee6b P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:49.675218 23957 tablet_bootstrap.cc:492] T fedbec8860904a2cb5294037b7a4bcad P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:49.676460 23955 tablet_bootstrap.cc:492] T 3be66f4b90824a21a5f41915c9041970 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:49.676970 23956 tablet_bootstrap.cc:492] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:49.679677 23957 tablet_bootstrap.cc:654] T fedbec8860904a2cb5294037b7a4bcad P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:49.679704 23955 tablet_bootstrap.cc:654] T 3be66f4b90824a21a5f41915c9041970 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:49.679716 23956 tablet_bootstrap.cc:654] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:49.680332 23954 tablet_bootstrap.cc:492] T d81f9cab671a4258aee1dd4ea4a5ee6b P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:49.680542 23954 ts_tablet_manager.cc:1367] T d81f9cab671a4258aee1dd4ea4a5ee6b P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240502 01:53:49.681692 23954 raft_consensus.cc:348] T d81f9cab671a4258aee1dd4ea4a5ee6b P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:49.682143 23954 raft_consensus.cc:374] T d81f9cab671a4258aee1dd4ea4a5ee6b P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.682322 23954 raft_consensus.cc:729] T d81f9cab671a4258aee1dd4ea4a5ee6b P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.682992 23954 consensus_queue.cc:260] T d81f9cab671a4258aee1dd4ea4a5ee6b P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:49.683995 23954 ts_tablet_manager.cc:1398] T d81f9cab671a4258aee1dd4ea4a5ee6b P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:49.687690 23956 tablet_bootstrap.cc:492] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:49.687930 23956 ts_tablet_manager.cc:1367] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.002s
I20240502 01:53:49.688843 23956 raft_consensus.cc:348] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:49.689152 23956 raft_consensus.cc:374] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.689291 23956 raft_consensus.cc:729] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.689621 23956 consensus_queue.cc:260] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:49.691247 23956 ts_tablet_manager.cc:1398] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:49.692910 23956 tablet_bootstrap.cc:492] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:49.694397 23954 tablet_bootstrap.cc:492] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:49.696044 23956 tablet_bootstrap.cc:654] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:49.698532 23955 tablet_bootstrap.cc:492] T 3be66f4b90824a21a5f41915c9041970 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:49.698734 23955 ts_tablet_manager.cc:1367] T 3be66f4b90824a21a5f41915c9041970 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.013s
I20240502 01:53:49.699163 23954 tablet_bootstrap.cc:654] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:49.699393 23957 tablet_bootstrap.cc:492] T fedbec8860904a2cb5294037b7a4bcad P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:49.699633 23957 ts_tablet_manager.cc:1367] T fedbec8860904a2cb5294037b7a4bcad P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.009s
I20240502 01:53:49.699721 23955 raft_consensus.cc:348] T 3be66f4b90824a21a5f41915c9041970 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:49.700034 23955 raft_consensus.cc:374] T 3be66f4b90824a21a5f41915c9041970 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.700191 23955 raft_consensus.cc:729] T 3be66f4b90824a21a5f41915c9041970 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.700533 23955 consensus_queue.cc:260] T 3be66f4b90824a21a5f41915c9041970 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:49.700728 23957 raft_consensus.cc:348] T fedbec8860904a2cb5294037b7a4bcad P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:49.701172 23957 raft_consensus.cc:374] T fedbec8860904a2cb5294037b7a4bcad P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.701313 23957 raft_consensus.cc:729] T fedbec8860904a2cb5294037b7a4bcad P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.701632 23957 consensus_queue.cc:260] T fedbec8860904a2cb5294037b7a4bcad P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:49.702067 23955 ts_tablet_manager.cc:1398] T 3be66f4b90824a21a5f41915c9041970 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:53:49.702368 23955 tablet_bootstrap.cc:492] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:49.702647 23957 ts_tablet_manager.cc:1398] T fedbec8860904a2cb5294037b7a4bcad P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:49.704069 23955 tablet_bootstrap.cc:654] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:49.704272 23956 tablet_bootstrap.cc:492] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:49.704564 23956 ts_tablet_manager.cc:1367] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20240502 01:53:49.705682 23956 raft_consensus.cc:348] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:49.706079 23956 raft_consensus.cc:374] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.706243 23956 raft_consensus.cc:729] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.706611 23956 consensus_queue.cc:260] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:49.707420 23956 ts_tablet_manager.cc:1398] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:49.707901 23956 tablet_bootstrap.cc:492] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:49.710343 23956 tablet_bootstrap.cc:654] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:49.712899 23954 tablet_bootstrap.cc:492] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:49.712975 23956 tablet_bootstrap.cc:492] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:49.713117 23956 ts_tablet_manager.cc:1367] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 01:53:49.713251 23954 ts_tablet_manager.cc:1367] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.009s
I20240502 01:53:49.713915 23956 raft_consensus.cc:348] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:49.714146 23956 raft_consensus.cc:374] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.714253 23956 raft_consensus.cc:729] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.714278 23954 raft_consensus.cc:348] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:49.714488 23956 consensus_queue.cc:260] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:49.714649 23954 raft_consensus.cc:374] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.714900 23954 raft_consensus.cc:729] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.715297 23956 ts_tablet_manager.cc:1398] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:49.715240 23954 consensus_queue.cc:260] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:49.715967 23954 ts_tablet_manager.cc:1398] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:49.716348 23954 tablet_bootstrap.cc:492] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:49.718230 23954 tablet_bootstrap.cc:654] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:49.721098 23955 tablet_bootstrap.cc:492] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:49.721290 23955 ts_tablet_manager.cc:1367] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.004s
I20240502 01:53:49.722175 23955 raft_consensus.cc:348] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:49.722661 23955 raft_consensus.cc:374] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.722769 23955 raft_consensus.cc:729] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.723017 23954 tablet_bootstrap.cc:492] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:49.723034 23955 consensus_queue.cc:260] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:49.723189 23954 ts_tablet_manager.cc:1367] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 01:53:49.723680 23955 ts_tablet_manager.cc:1398] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:49.724234 23954 raft_consensus.cc:348] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:49.724468 23954 raft_consensus.cc:374] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.724594 23954 raft_consensus.cc:729] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.725075 23954 consensus_queue.cc:260] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:49.725811 23954 ts_tablet_manager.cc:1398] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:49.840692 23803 raft_consensus.cc:482] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:49.840934 23803 raft_consensus.cc:504] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:49.841585 23803 leader_election.cc:288] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:49.842227 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fedbec8860904a2cb5294037b7a4bcad" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:49.842242 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fedbec8860904a2cb5294037b7a4bcad" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:49.842497 21857 raft_consensus.cc:2454] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:49.842494 21986 raft_consensus.cc:2454] T fedbec8860904a2cb5294037b7a4bcad P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:49.843189 22053 leader_election.cc:302] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:49.843495 23803 raft_consensus.cc:2781] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:49.843633 23803 raft_consensus.cc:482] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:49.843756 23803 raft_consensus.cc:3037] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.846114 23803 raft_consensus.cc:504] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:49.846773 23803 leader_election.cc:288] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:49.847199 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fedbec8860904a2cb5294037b7a4bcad" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:49.847445 21856 raft_consensus.cc:3037] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.847443 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fedbec8860904a2cb5294037b7a4bcad" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:49.847645 21986 raft_consensus.cc:3037] T fedbec8860904a2cb5294037b7a4bcad P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.850677 21986 raft_consensus.cc:2454] T fedbec8860904a2cb5294037b7a4bcad P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:49.850695 21856 raft_consensus.cc:2454] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:49.851300 22053 leader_election.cc:302] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:49.851665 23874 raft_consensus.cc:2781] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:49.851850 23874 raft_consensus.cc:686] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:53:49.852238 23874 consensus_queue.cc:237] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:49.856220 21712 catalog_manager.cc:5484] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:53:49.884761 23769 raft_consensus.cc:482] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:49.885046 23769 raft_consensus.cc:504] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:49.885962 23769 leader_election.cc:288] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:49.886587 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81f9cab671a4258aee1dd4ea4a5ee6b" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:49.886592 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81f9cab671a4258aee1dd4ea4a5ee6b" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:49.886873 22119 raft_consensus.cc:2454] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:49.886915 21856 raft_consensus.cc:2454] T d81f9cab671a4258aee1dd4ea4a5ee6b P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:49.887401 22184 leader_election.cc:302] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:49.887769 23769 raft_consensus.cc:2781] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:49.887887 23769 raft_consensus.cc:482] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:49.888015 23769 raft_consensus.cc:3037] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.890029 23769 raft_consensus.cc:504] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:49.890733 23769 leader_election.cc:288] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:49.891041 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81f9cab671a4258aee1dd4ea4a5ee6b" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:49.891264 21857 raft_consensus.cc:3037] T d81f9cab671a4258aee1dd4ea4a5ee6b P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.891255 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81f9cab671a4258aee1dd4ea4a5ee6b" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:49.891516 22119 raft_consensus.cc:3037] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.893784 21857 raft_consensus.cc:2454] T d81f9cab671a4258aee1dd4ea4a5ee6b P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:49.893955 22119 raft_consensus.cc:2454] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:49.894191 22184 leader_election.cc:302] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:49.894517 23769 raft_consensus.cc:2781] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:49.894706 23769 raft_consensus.cc:686] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:53:49.895246 23769 consensus_queue.cc:237] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:49.898749 21712 catalog_manager.cc:5484] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:53:49.951007 23919 raft_consensus.cc:482] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:49.951225 23919 raft_consensus.cc:504] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:49.952340 23919 leader_election.cc:288] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:49.952759 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be66f4b90824a21a5f41915c9041970" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:49.953022 22119 raft_consensus.cc:2454] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:49.953073 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be66f4b90824a21a5f41915c9041970" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:49.953306 22251 raft_consensus.cc:2454] T 3be66f4b90824a21a5f41915c9041970 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:49.953441 21792 leader_election.cc:302] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:49.953752 23919 raft_consensus.cc:2781] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:49.953886 23919 raft_consensus.cc:482] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:49.953980 23919 raft_consensus.cc:3037] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.956552 23919 raft_consensus.cc:504] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:49.957180 23919 leader_election.cc:288] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:49.957530 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be66f4b90824a21a5f41915c9041970" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:49.957639 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be66f4b90824a21a5f41915c9041970" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:49.957713 22119 raft_consensus.cc:3037] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.957842 22251 raft_consensus.cc:3037] T 3be66f4b90824a21a5f41915c9041970 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.959589 22119 raft_consensus.cc:2454] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:49.959662 22251 raft_consensus.cc:2454] T 3be66f4b90824a21a5f41915c9041970 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:49.960028 21792 leader_election.cc:302] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:49.960335 23919 raft_consensus.cc:2781] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:49.960494 23919 raft_consensus.cc:686] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:53:49.960784 23919 consensus_queue.cc:237] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:49.964299 21712 catalog_manager.cc:5484] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:53:50.268312 23874 consensus_queue.cc:1035] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:50.286216 23935 consensus_queue.cc:1035] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:50.293968 23967 consensus_queue.cc:1035] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:50.303670 23769 consensus_queue.cc:1035] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:50.354811 23971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:50.355139 23971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:50.362901 23971 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:53:50.367807 23919 consensus_queue.cc:1035] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:50.375524 23858 consensus_queue.cc:1035] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_13'
INSERT report
rows total: 2000
time total: 215.01 ms
time per row: 0.107505 ms
I20240502 01:53:50.746624 21711 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:50.756376 22230 tablet_service.cc:1467] Processing CreateTablet for tablet c6fa6da27fc745e79963ffa294472d9e (DEFAULT_TABLE table=rebalance_test_table_14 [id=27c9bf24f886426b92b80b01e4a8d669]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:50.756558 21969 tablet_service.cc:1467] Processing CreateTablet for tablet c6fa6da27fc745e79963ffa294472d9e (DEFAULT_TABLE table=rebalance_test_table_14 [id=27c9bf24f886426b92b80b01e4a8d669]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:50.756958 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6fa6da27fc745e79963ffa294472d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.757058 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6fa6da27fc745e79963ffa294472d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.757123 21837 tablet_service.cc:1467] Processing CreateTablet for tablet c6fa6da27fc745e79963ffa294472d9e (DEFAULT_TABLE table=rebalance_test_table_14 [id=27c9bf24f886426b92b80b01e4a8d669]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:50.757380 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 68e1096b01964772ac02755f1ae21c53 (DEFAULT_TABLE table=rebalance_test_table_14 [id=27c9bf24f886426b92b80b01e4a8d669]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:50.757583 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6fa6da27fc745e79963ffa294472d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.757817 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e1096b01964772ac02755f1ae21c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.765071 21967 tablet_service.cc:1467] Processing CreateTablet for tablet 68e1096b01964772ac02755f1ae21c53 (DEFAULT_TABLE table=rebalance_test_table_14 [id=27c9bf24f886426b92b80b01e4a8d669]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:50.765633 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e1096b01964772ac02755f1ae21c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.766302 21835 tablet_service.cc:1467] Processing CreateTablet for tablet 270e238486f24381815427c4ee2d587a (DEFAULT_TABLE table=rebalance_test_table_14 [id=27c9bf24f886426b92b80b01e4a8d669]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:50.766698 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270e238486f24381815427c4ee2d587a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.769876 22229 tablet_service.cc:1467] Processing CreateTablet for tablet 68e1096b01964772ac02755f1ae21c53 (DEFAULT_TABLE table=rebalance_test_table_14 [id=27c9bf24f886426b92b80b01e4a8d669]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:50.770153 21968 tablet_service.cc:1467] Processing CreateTablet for tablet 270e238486f24381815427c4ee2d587a (DEFAULT_TABLE table=rebalance_test_table_14 [id=27c9bf24f886426b92b80b01e4a8d669]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:50.770438 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e1096b01964772ac02755f1ae21c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.770522 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270e238486f24381815427c4ee2d587a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.775312 24034 tablet_bootstrap.cc:492] T c6fa6da27fc745e79963ffa294472d9e P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:50.777333 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 270e238486f24381815427c4ee2d587a (DEFAULT_TABLE table=rebalance_test_table_14 [id=27c9bf24f886426b92b80b01e4a8d669]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:50.777861 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270e238486f24381815427c4ee2d587a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.780481 24034 tablet_bootstrap.cc:654] T c6fa6da27fc745e79963ffa294472d9e P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.786943 24035 tablet_bootstrap.cc:492] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:50.789798 24035 tablet_bootstrap.cc:654] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.790452 24036 tablet_bootstrap.cc:492] T 68e1096b01964772ac02755f1ae21c53 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:50.790931 24033 tablet_bootstrap.cc:492] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:50.792977 24033 tablet_bootstrap.cc:654] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.793478 24036 tablet_bootstrap.cc:654] T 68e1096b01964772ac02755f1ae21c53 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.799259 24034 tablet_bootstrap.cc:492] T c6fa6da27fc745e79963ffa294472d9e P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:50.799392 24036 tablet_bootstrap.cc:492] T 68e1096b01964772ac02755f1ae21c53 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:50.799547 24036 ts_tablet_manager.cc:1367] T 68e1096b01964772ac02755f1ae21c53 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 01:53:50.799700 24034 ts_tablet_manager.cc:1367] T c6fa6da27fc745e79963ffa294472d9e P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.003s
I20240502 01:53:50.800429 24036 raft_consensus.cc:348] T 68e1096b01964772ac02755f1ae21c53 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:50.800655 24036 raft_consensus.cc:374] T 68e1096b01964772ac02755f1ae21c53 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.800760 24036 raft_consensus.cc:729] T 68e1096b01964772ac02755f1ae21c53 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.801019 24036 consensus_queue.cc:260] T 68e1096b01964772ac02755f1ae21c53 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:50.801697 24036 ts_tablet_manager.cc:1398] T 68e1096b01964772ac02755f1ae21c53 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:50.801649 24034 raft_consensus.cc:348] T c6fa6da27fc745e79963ffa294472d9e P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:50.801983 24034 raft_consensus.cc:374] T c6fa6da27fc745e79963ffa294472d9e P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.802138 24034 raft_consensus.cc:729] T c6fa6da27fc745e79963ffa294472d9e P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.802248 24036 tablet_bootstrap.cc:492] T 270e238486f24381815427c4ee2d587a P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:50.802498 24034 consensus_queue.cc:260] T c6fa6da27fc745e79963ffa294472d9e P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:50.803472 24034 ts_tablet_manager.cc:1398] T c6fa6da27fc745e79963ffa294472d9e P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:53:50.803836 24034 tablet_bootstrap.cc:492] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:50.804206 24036 tablet_bootstrap.cc:654] T 270e238486f24381815427c4ee2d587a P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.806507 24034 tablet_bootstrap.cc:654] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.806807 24035 tablet_bootstrap.cc:492] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:50.807018 24035 ts_tablet_manager.cc:1367] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.008s
I20240502 01:53:50.808061 24035 raft_consensus.cc:348] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:50.808341 24035 raft_consensus.cc:374] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.808450 24035 raft_consensus.cc:729] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.808739 24035 consensus_queue.cc:260] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:50.809574 24035 ts_tablet_manager.cc:1398] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:50.812805 24033 tablet_bootstrap.cc:492] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:50.813472 24033 ts_tablet_manager.cc:1367] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.006s
I20240502 01:53:50.821856 24033 raft_consensus.cc:348] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:50.822131 24033 raft_consensus.cc:374] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.822232 24033 raft_consensus.cc:729] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.822500 24033 consensus_queue.cc:260] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:50.823261 24033 ts_tablet_manager.cc:1398] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:50.825063 24036 tablet_bootstrap.cc:492] T 270e238486f24381815427c4ee2d587a P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:50.825313 24036 ts_tablet_manager.cc:1367] T 270e238486f24381815427c4ee2d587a P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.011s
I20240502 01:53:50.826309 24036 raft_consensus.cc:348] T 270e238486f24381815427c4ee2d587a P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:50.826637 24036 raft_consensus.cc:374] T 270e238486f24381815427c4ee2d587a P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.826737 24036 raft_consensus.cc:729] T 270e238486f24381815427c4ee2d587a P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.827060 24036 consensus_queue.cc:260] T 270e238486f24381815427c4ee2d587a P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:50.827802 24036 ts_tablet_manager.cc:1398] T 270e238486f24381815427c4ee2d587a P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:50.832352 24036 tablet_bootstrap.cc:492] T c6fa6da27fc745e79963ffa294472d9e P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:50.837921 24036 tablet_bootstrap.cc:654] T c6fa6da27fc745e79963ffa294472d9e P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.839399 24033 tablet_bootstrap.cc:492] T 68e1096b01964772ac02755f1ae21c53 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:50.841776 24033 tablet_bootstrap.cc:654] T 68e1096b01964772ac02755f1ae21c53 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.843951 24034 tablet_bootstrap.cc:492] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:50.844142 24034 ts_tablet_manager.cc:1367] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.040s user 0.016s sys 0.010s
I20240502 01:53:50.844944 24034 raft_consensus.cc:348] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:50.845188 24034 raft_consensus.cc:374] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.845265 24034 raft_consensus.cc:729] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.845517 24034 consensus_queue.cc:260] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:50.847115 24034 ts_tablet_manager.cc:1398] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:50.854462 24036 tablet_bootstrap.cc:492] T c6fa6da27fc745e79963ffa294472d9e P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:50.854671 24036 ts_tablet_manager.cc:1367] T c6fa6da27fc745e79963ffa294472d9e P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.008s
I20240502 01:53:50.855326 24033 tablet_bootstrap.cc:492] T 68e1096b01964772ac02755f1ae21c53 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:50.855523 24033 ts_tablet_manager.cc:1367] T 68e1096b01964772ac02755f1ae21c53 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20240502 01:53:50.855460 24036 raft_consensus.cc:348] T c6fa6da27fc745e79963ffa294472d9e P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:50.855712 24036 raft_consensus.cc:374] T c6fa6da27fc745e79963ffa294472d9e P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.855809 24036 raft_consensus.cc:729] T c6fa6da27fc745e79963ffa294472d9e P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.856048 24036 consensus_queue.cc:260] T c6fa6da27fc745e79963ffa294472d9e P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:50.856627 24036 ts_tablet_manager.cc:1398] T c6fa6da27fc745e79963ffa294472d9e P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:50.856606 24033 raft_consensus.cc:348] T 68e1096b01964772ac02755f1ae21c53 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:50.856854 24033 raft_consensus.cc:374] T 68e1096b01964772ac02755f1ae21c53 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.856969 24033 raft_consensus.cc:729] T 68e1096b01964772ac02755f1ae21c53 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.857282 24033 consensus_queue.cc:260] T 68e1096b01964772ac02755f1ae21c53 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:50.858045 24033 ts_tablet_manager.cc:1398] T 68e1096b01964772ac02755f1ae21c53 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:50.859390 24033 tablet_bootstrap.cc:492] T 270e238486f24381815427c4ee2d587a P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:50.863649 24033 tablet_bootstrap.cc:654] T 270e238486f24381815427c4ee2d587a P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.866443 24033 tablet_bootstrap.cc:492] T 270e238486f24381815427c4ee2d587a P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:50.866602 24033 ts_tablet_manager.cc:1367] T 270e238486f24381815427c4ee2d587a P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20240502 01:53:50.867334 24033 raft_consensus.cc:348] T 270e238486f24381815427c4ee2d587a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:50.867547 24033 raft_consensus.cc:374] T 270e238486f24381815427c4ee2d587a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.867630 24033 raft_consensus.cc:729] T 270e238486f24381815427c4ee2d587a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.867856 24033 consensus_queue.cc:260] T 270e238486f24381815427c4ee2d587a P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:50.868420 24033 ts_tablet_manager.cc:1398] T 270e238486f24381815427c4ee2d587a P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:50.902330 23769 raft_consensus.cc:482] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:50.902542 23769 raft_consensus.cc:504] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:50.903484 23769 leader_election.cc:288] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:50.903985 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6fa6da27fc745e79963ffa294472d9e" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:50.904050 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6fa6da27fc745e79963ffa294472d9e" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:50.904269 21986 raft_consensus.cc:2454] T c6fa6da27fc745e79963ffa294472d9e P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:50.904328 21856 raft_consensus.cc:2454] T c6fa6da27fc745e79963ffa294472d9e P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:50.904747 22184 leader_election.cc:302] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:50.905104 23769 raft_consensus.cc:2781] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:50.905213 23769 raft_consensus.cc:482] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:50.905321 23769 raft_consensus.cc:3037] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:50.907361 23769 raft_consensus.cc:504] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:50.907907 23769 leader_election.cc:288] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:50.908421 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6fa6da27fc745e79963ffa294472d9e" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:50.908478 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6fa6da27fc745e79963ffa294472d9e" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:50.908679 21986 raft_consensus.cc:3037] T c6fa6da27fc745e79963ffa294472d9e P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:50.908716 21856 raft_consensus.cc:3037] T c6fa6da27fc745e79963ffa294472d9e P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:50.911027 21986 raft_consensus.cc:2454] T c6fa6da27fc745e79963ffa294472d9e P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:50.911018 21856 raft_consensus.cc:2454] T c6fa6da27fc745e79963ffa294472d9e P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:50.911401 22184 leader_election.cc:302] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:50.911713 23769 raft_consensus.cc:2781] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:50.911895 23769 raft_consensus.cc:686] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:53:50.912143 23769 consensus_queue.cc:237] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:50.915136 21714 catalog_manager.cc:5484] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:53:51.103399 23967 raft_consensus.cc:482] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:51.103628 23967 raft_consensus.cc:504] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:51.104480 23967 leader_election.cc:288] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:51.104959 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e1096b01964772ac02755f1ae21c53" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:51.104993 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e1096b01964772ac02755f1ae21c53" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:51.105250 22251 raft_consensus.cc:2454] T 68e1096b01964772ac02755f1ae21c53 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:51.105227 21988 raft_consensus.cc:2454] T 68e1096b01964772ac02755f1ae21c53 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:51.105723 22053 leader_election.cc:302] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:51.106012 23967 raft_consensus.cc:2781] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:51.106122 23967 raft_consensus.cc:482] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:51.106249 23967 raft_consensus.cc:3037] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.108283 23967 raft_consensus.cc:504] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:51.108909 23967 leader_election.cc:288] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:51.109282 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e1096b01964772ac02755f1ae21c53" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:51.109375 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e1096b01964772ac02755f1ae21c53" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:51.109534 21988 raft_consensus.cc:3037] T 68e1096b01964772ac02755f1ae21c53 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.109588 22251 raft_consensus.cc:3037] T 68e1096b01964772ac02755f1ae21c53 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.112430 22251 raft_consensus.cc:2454] T 68e1096b01964772ac02755f1ae21c53 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:51.112444 21988 raft_consensus.cc:2454] T 68e1096b01964772ac02755f1ae21c53 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:51.112864 22055 leader_election.cc:302] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:51.113211 23967 raft_consensus.cc:2781] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:51.113386 23967 raft_consensus.cc:686] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:53:51.113672 23967 consensus_queue.cc:237] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:51.117147 21714 catalog_manager.cc:5484] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:53:51.197685 23858 raft_consensus.cc:482] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:51.197906 23858 raft_consensus.cc:504] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:51.198637 23858 leader_election.cc:288] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:51.199296 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270e238486f24381815427c4ee2d587a" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:51.199347 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270e238486f24381815427c4ee2d587a" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:51.199586 22251 raft_consensus.cc:2454] T 270e238486f24381815427c4ee2d587a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:51.199586 21988 raft_consensus.cc:2454] T 270e238486f24381815427c4ee2d587a P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:51.200060 21792 leader_election.cc:302] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:51.200369 23858 raft_consensus.cc:2781] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:51.200472 23858 raft_consensus.cc:482] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:51.200592 23858 raft_consensus.cc:3037] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.202936 23858 raft_consensus.cc:504] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:51.203538 23858 leader_election.cc:288] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:51.203902 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270e238486f24381815427c4ee2d587a" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:51.203966 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270e238486f24381815427c4ee2d587a" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:51.204130 21988 raft_consensus.cc:3037] T 270e238486f24381815427c4ee2d587a P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.204176 22251 raft_consensus.cc:3037] T 270e238486f24381815427c4ee2d587a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.206261 22251 raft_consensus.cc:2454] T 270e238486f24381815427c4ee2d587a P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:51.206310 21988 raft_consensus.cc:2454] T 270e238486f24381815427c4ee2d587a P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:51.206725 21790 leader_election.cc:302] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:53:51.206991 23858 raft_consensus.cc:2781] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:51.207147 23858 raft_consensus.cc:686] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:53:51.207405 23858 consensus_queue.cc:237] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:51.211288 21714 catalog_manager.cc:5484] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:53:51.318961 23769 consensus_queue.cc:1035] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:51.326999 23769 consensus_queue.cc:1035] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:51.556851 23874 consensus_queue.cc:1035] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:51.562696 23874 consensus_queue.cc:1035] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:51.598592 24050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:51.599009 24050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:53:51.599663 24048 consensus_queue.cc:1035] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:51.604997 23858 consensus_queue.cc:1035] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:51.610040 24050 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
W20240502 01:53:51.940447 22055 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.196:43093, user_credentials={real_user=slave}} blocked reactor thread for 55416.6us
INSERT report
rows total: 2000
time total: 300.701 ms
time per row: 0.150351 ms
I20240502 01:53:52.124133 21708 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:52.133980 21967 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3f40a9f1c34644b787905c28a54b29 (DEFAULT_TABLE table=rebalance_test_table_15 [id=855af8637536470bb3f01aeac50e07e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:52.134097 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3f40a9f1c34644b787905c28a54b29 (DEFAULT_TABLE table=rebalance_test_table_15 [id=855af8637536470bb3f01aeac50e07e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:52.134552 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3f40a9f1c34644b787905c28a54b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.134544 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3f40a9f1c34644b787905c28a54b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.135047 21969 tablet_service.cc:1467] Processing CreateTablet for tablet de3eb3ed981d4f629eaa7f473ea3b430 (DEFAULT_TABLE table=rebalance_test_table_15 [id=855af8637536470bb3f01aeac50e07e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:52.135370 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3eb3ed981d4f629eaa7f473ea3b430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.137590 22098 tablet_service.cc:1467] Processing CreateTablet for tablet de3eb3ed981d4f629eaa7f473ea3b430 (DEFAULT_TABLE table=rebalance_test_table_15 [id=855af8637536470bb3f01aeac50e07e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:52.137637 21837 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3f40a9f1c34644b787905c28a54b29 (DEFAULT_TABLE table=rebalance_test_table_15 [id=855af8637536470bb3f01aeac50e07e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:52.138051 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3eb3ed981d4f629eaa7f473ea3b430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.138090 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3f40a9f1c34644b787905c28a54b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.143289 22231 tablet_service.cc:1467] Processing CreateTablet for tablet daaadb773202454894890c3379c4a58d (DEFAULT_TABLE table=rebalance_test_table_15 [id=855af8637536470bb3f01aeac50e07e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:52.143764 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daaadb773202454894890c3379c4a58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.144366 21835 tablet_service.cc:1467] Processing CreateTablet for tablet daaadb773202454894890c3379c4a58d (DEFAULT_TABLE table=rebalance_test_table_15 [id=855af8637536470bb3f01aeac50e07e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:52.144770 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daaadb773202454894890c3379c4a58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.149549 22099 tablet_service.cc:1467] Processing CreateTablet for tablet daaadb773202454894890c3379c4a58d (DEFAULT_TABLE table=rebalance_test_table_15 [id=855af8637536470bb3f01aeac50e07e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:52.150156 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daaadb773202454894890c3379c4a58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.153355 24114 tablet_bootstrap.cc:492] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:52.153369 22230 tablet_service.cc:1467] Processing CreateTablet for tablet de3eb3ed981d4f629eaa7f473ea3b430 (DEFAULT_TABLE table=rebalance_test_table_15 [id=855af8637536470bb3f01aeac50e07e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:52.153832 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3eb3ed981d4f629eaa7f473ea3b430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.154170 24113 tablet_bootstrap.cc:492] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:52.155263 24114 tablet_bootstrap.cc:654] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.155968 24113 tablet_bootstrap.cc:654] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.158056 24114 tablet_bootstrap.cc:492] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:52.158515 24114 ts_tablet_manager.cc:1367] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 01:53:52.159094 24115 tablet_bootstrap.cc:492] T de3eb3ed981d4f629eaa7f473ea3b430 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:52.159698 24114 raft_consensus.cc:348] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:52.160115 24114 raft_consensus.cc:374] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.160280 24114 raft_consensus.cc:729] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.160661 24114 consensus_queue.cc:260] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:52.161480 24115 tablet_bootstrap.cc:654] T de3eb3ed981d4f629eaa7f473ea3b430 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.161595 24114 ts_tablet_manager.cc:1398] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:52.163583 24113 tablet_bootstrap.cc:492] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:52.164345 24113 ts_tablet_manager.cc:1367] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 01:53:52.166618 24115 tablet_bootstrap.cc:492] T de3eb3ed981d4f629eaa7f473ea3b430 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:52.167081 24115 ts_tablet_manager.cc:1367] T de3eb3ed981d4f629eaa7f473ea3b430 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 01:53:52.167974 24115 raft_consensus.cc:348] T de3eb3ed981d4f629eaa7f473ea3b430 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:52.168255 24115 raft_consensus.cc:374] T de3eb3ed981d4f629eaa7f473ea3b430 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.168365 24115 raft_consensus.cc:729] T de3eb3ed981d4f629eaa7f473ea3b430 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.168608 24115 consensus_queue.cc:260] T de3eb3ed981d4f629eaa7f473ea3b430 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:52.169325 24115 ts_tablet_manager.cc:1398] T de3eb3ed981d4f629eaa7f473ea3b430 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:52.169720 24115 tablet_bootstrap.cc:492] T 5b3f40a9f1c34644b787905c28a54b29 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:52.170267 24113 raft_consensus.cc:348] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:52.170658 24114 tablet_bootstrap.cc:492] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:52.171288 24113 raft_consensus.cc:374] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.171576 24113 raft_consensus.cc:729] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.171844 24115 tablet_bootstrap.cc:654] T 5b3f40a9f1c34644b787905c28a54b29 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.172209 24113 consensus_queue.cc:260] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:52.172389 24114 tablet_bootstrap.cc:654] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.173449 24113 ts_tablet_manager.cc:1398] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:53:52.176118 24113 tablet_bootstrap.cc:492] T 5b3f40a9f1c34644b787905c28a54b29 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:52.182952 24113 tablet_bootstrap.cc:654] T 5b3f40a9f1c34644b787905c28a54b29 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.186678 24116 tablet_bootstrap.cc:492] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:52.188311 24115 tablet_bootstrap.cc:492] T 5b3f40a9f1c34644b787905c28a54b29 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:52.188333 24114 tablet_bootstrap.cc:492] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:52.188468 24113 tablet_bootstrap.cc:492] T 5b3f40a9f1c34644b787905c28a54b29 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:52.188526 24115 ts_tablet_manager.cc:1367] T 5b3f40a9f1c34644b787905c28a54b29 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.004s
I20240502 01:53:52.188529 24114 ts_tablet_manager.cc:1367] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.005s
I20240502 01:53:52.188779 24113 ts_tablet_manager.cc:1367] T 5b3f40a9f1c34644b787905c28a54b29 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240502 01:53:52.189319 24116 tablet_bootstrap.cc:654] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.189464 24114 raft_consensus.cc:348] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:52.189489 24115 raft_consensus.cc:348] T 5b3f40a9f1c34644b787905c28a54b29 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:52.189741 24115 raft_consensus.cc:374] T 5b3f40a9f1c34644b787905c28a54b29 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.189742 24114 raft_consensus.cc:374] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.189644 24113 raft_consensus.cc:348] T 5b3f40a9f1c34644b787905c28a54b29 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:52.189827 24114 raft_consensus.cc:729] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.189826 24115 raft_consensus.cc:729] T 5b3f40a9f1c34644b787905c28a54b29 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.189869 24113 raft_consensus.cc:374] T 5b3f40a9f1c34644b787905c28a54b29 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.190018 24113 raft_consensus.cc:729] T 5b3f40a9f1c34644b787905c28a54b29 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.190151 24115 consensus_queue.cc:260] T 5b3f40a9f1c34644b787905c28a54b29 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:52.190165 24114 consensus_queue.cc:260] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:52.190477 24113 consensus_queue.cc:260] T 5b3f40a9f1c34644b787905c28a54b29 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:52.191416 24113 ts_tablet_manager.cc:1398] T 5b3f40a9f1c34644b787905c28a54b29 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:52.192711 24114 ts_tablet_manager.cc:1398] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:52.194904 24115 ts_tablet_manager.cc:1398] T 5b3f40a9f1c34644b787905c28a54b29 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 01:53:52.195298 24113 tablet_bootstrap.cc:492] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:52.197173 24113 tablet_bootstrap.cc:654] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.198479 24116 tablet_bootstrap.cc:492] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:52.198691 24116 ts_tablet_manager.cc:1367] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240502 01:53:52.200317 24116 raft_consensus.cc:348] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:52.200932 24116 raft_consensus.cc:374] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.201050 24116 raft_consensus.cc:729] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.201347 24116 consensus_queue.cc:260] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:52.202136 24116 ts_tablet_manager.cc:1398] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:52.202512 24116 tablet_bootstrap.cc:492] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:52.204457 24116 tablet_bootstrap.cc:654] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.205188 24113 tablet_bootstrap.cc:492] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:52.205350 24113 ts_tablet_manager.cc:1367] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20240502 01:53:52.206449 24113 raft_consensus.cc:348] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:52.206671 24113 raft_consensus.cc:374] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.206761 24113 raft_consensus.cc:729] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.207054 24116 tablet_bootstrap.cc:492] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:52.207007 24113 consensus_queue.cc:260] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:52.207229 24116 ts_tablet_manager.cc:1367] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 01:53:52.207792 24113 ts_tablet_manager.cc:1398] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:52.208237 24116 raft_consensus.cc:348] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:52.208456 24116 raft_consensus.cc:374] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.208647 24116 raft_consensus.cc:729] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.208925 24116 consensus_queue.cc:260] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:52.209522 24116 ts_tablet_manager.cc:1398] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:52.235461 24104 raft_consensus.cc:482] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:52.235721 24104 raft_consensus.cc:504] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:52.236653 24104 leader_election.cc:288] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:52.237074 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3eb3ed981d4f629eaa7f473ea3b430" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:52.237365 22119 raft_consensus.cc:2454] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:52.237567 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3eb3ed981d4f629eaa7f473ea3b430" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:52.237818 21986 raft_consensus.cc:2454] T de3eb3ed981d4f629eaa7f473ea3b430 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:52.237891 22186 leader_election.cc:302] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:52.238178 24104 raft_consensus.cc:2781] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:52.238313 24104 raft_consensus.cc:482] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:52.238409 24104 raft_consensus.cc:3037] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:52.241400 24104 raft_consensus.cc:504] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:52.242556 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3eb3ed981d4f629eaa7f473ea3b430" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:52.242825 22119 raft_consensus.cc:3037] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:52.242836 24104 leader_election.cc:288] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:52.243386 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3eb3ed981d4f629eaa7f473ea3b430" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:52.243657 21986 raft_consensus.cc:3037] T de3eb3ed981d4f629eaa7f473ea3b430 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:52.245071 22119 raft_consensus.cc:2454] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:52.245589 22186 leader_election.cc:302] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:52.246315 24104 raft_consensus.cc:2781] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:52.246471 24104 raft_consensus.cc:686] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:53:52.246852 24104 consensus_queue.cc:237] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:52.247527 21986 raft_consensus.cc:2454] T de3eb3ed981d4f629eaa7f473ea3b430 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:52.251132 21714 catalog_manager.cc:5484] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:53:52.295068 23967 raft_consensus.cc:482] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:52.295289 23967 raft_consensus.cc:504] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:52.296077 23967 leader_election.cc:288] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:52.296604 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daaadb773202454894890c3379c4a58d" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:52.296768 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daaadb773202454894890c3379c4a58d" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:52.296854 21856 raft_consensus.cc:2454] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:52.297037 22250 raft_consensus.cc:2454] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:52.297372 22053 leader_election.cc:302] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:52.297735 23967 raft_consensus.cc:2781] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:52.297860 23967 raft_consensus.cc:482] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:52.297953 23967 raft_consensus.cc:3037] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:52.300169 23967 raft_consensus.cc:504] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:52.301018 23967 leader_election.cc:288] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:52.301292 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daaadb773202454894890c3379c4a58d" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:52.301314 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daaadb773202454894890c3379c4a58d" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:52.301463 21856 raft_consensus.cc:3037] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:52.301518 22250 raft_consensus.cc:3037] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:52.303442 21856 raft_consensus.cc:2454] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:52.303897 22053 leader_election.cc:302] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:52.304255 22250 raft_consensus.cc:2454] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:52.304427 23967 raft_consensus.cc:2781] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:52.305145 23967 raft_consensus.cc:686] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:53:52.305480 23967 consensus_queue.cc:237] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:52.309949 21714 catalog_manager.cc:5484] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:53:52.491768 24091 raft_consensus.cc:482] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:52.491958 24091 raft_consensus.cc:504] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:52.492643 24091 leader_election.cc:288] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:52.493156 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3f40a9f1c34644b787905c28a54b29" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:52.493247 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3f40a9f1c34644b787905c28a54b29" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:52.493427 21986 raft_consensus.cc:2454] T 5b3f40a9f1c34644b787905c28a54b29 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:52.493496 22119 raft_consensus.cc:2454] T 5b3f40a9f1c34644b787905c28a54b29 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:52.493894 21790 leader_election.cc:302] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:53:52.494192 24091 raft_consensus.cc:2781] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:52.494297 24091 raft_consensus.cc:482] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:52.494400 24091 raft_consensus.cc:3037] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:52.496537 24091 raft_consensus.cc:504] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:52.497188 24091 leader_election.cc:288] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:52.497601 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3f40a9f1c34644b787905c28a54b29" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:52.497704 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3f40a9f1c34644b787905c28a54b29" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:52.497808 21986 raft_consensus.cc:3037] T 5b3f40a9f1c34644b787905c28a54b29 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:52.497922 22119 raft_consensus.cc:3037] T 5b3f40a9f1c34644b787905c28a54b29 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:52.499766 21986 raft_consensus.cc:2454] T 5b3f40a9f1c34644b787905c28a54b29 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:52.499805 22119 raft_consensus.cc:2454] T 5b3f40a9f1c34644b787905c28a54b29 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:52.500223 21790 leader_election.cc:302] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:52.500586 24091 raft_consensus.cc:2781] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:52.501025 24091 raft_consensus.cc:686] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:53:52.501358 24091 consensus_queue.cc:237] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:52.504810 21714 catalog_manager.cc:5484] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:53:52.667296 24104 consensus_queue.cc:1035] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:52.671883 24104 consensus_queue.cc:1035] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:52.736274 23967 consensus_queue.cc:1035] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:52.741587 24103 consensus_queue.cc:1035] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:52.918829 24130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:52.919200 24130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:53:52.924444 24091 consensus_queue.cc:1035] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:52.928113 24130 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:53:52.930354 24128 consensus_queue.cc:1035] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_15'
W20240502 01:53:53.197510 21792 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.195:38879, user_credentials={real_user=slave}} blocked reactor thread for 51333.5us
INSERT report
rows total: 2000
time total: 233.179 ms
time per row: 0.11659 ms
I20240502 01:53:53.327425 21713 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:53.337100 21969 tablet_service.cc:1467] Processing CreateTablet for tablet bdb710e59756492d870d1a152a629094 (DEFAULT_TABLE table=rebalance_test_table_16 [id=efd69cb962724c4f85c2825debc9eba0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:53.337512 22231 tablet_service.cc:1467] Processing CreateTablet for tablet bdb710e59756492d870d1a152a629094 (DEFAULT_TABLE table=rebalance_test_table_16 [id=efd69cb962724c4f85c2825debc9eba0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:53.337652 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb710e59756492d870d1a152a629094. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.337909 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb710e59756492d870d1a152a629094. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.338550 21837 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1686dbe93c42b597f32422b6e8ac75 (DEFAULT_TABLE table=rebalance_test_table_16 [id=efd69cb962724c4f85c2825debc9eba0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:53.338973 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1686dbe93c42b597f32422b6e8ac75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.341729 22099 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1686dbe93c42b597f32422b6e8ac75 (DEFAULT_TABLE table=rebalance_test_table_16 [id=efd69cb962724c4f85c2825debc9eba0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:53.341782 22098 tablet_service.cc:1467] Processing CreateTablet for tablet b993ed799224421fa6fc29c4cf6dfbd0 (DEFAULT_TABLE table=rebalance_test_table_16 [id=efd69cb962724c4f85c2825debc9eba0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:53.341785 21835 tablet_service.cc:1467] Processing CreateTablet for tablet bdb710e59756492d870d1a152a629094 (DEFAULT_TABLE table=rebalance_test_table_16 [id=efd69cb962724c4f85c2825debc9eba0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:53.342171 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1686dbe93c42b597f32422b6e8ac75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.342201 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb710e59756492d870d1a152a629094. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.342653 22230 tablet_service.cc:1467] Processing CreateTablet for tablet b993ed799224421fa6fc29c4cf6dfbd0 (DEFAULT_TABLE table=rebalance_test_table_16 [id=efd69cb962724c4f85c2825debc9eba0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:53.343034 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b993ed799224421fa6fc29c4cf6dfbd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.346580 21967 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1686dbe93c42b597f32422b6e8ac75 (DEFAULT_TABLE table=rebalance_test_table_16 [id=efd69cb962724c4f85c2825debc9eba0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:53.347038 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1686dbe93c42b597f32422b6e8ac75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.353266 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b993ed799224421fa6fc29c4cf6dfbd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.353593 21968 tablet_service.cc:1467] Processing CreateTablet for tablet b993ed799224421fa6fc29c4cf6dfbd0 (DEFAULT_TABLE table=rebalance_test_table_16 [id=efd69cb962724c4f85c2825debc9eba0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:53.354120 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b993ed799224421fa6fc29c4cf6dfbd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.363850 24195 tablet_bootstrap.cc:492] T 6f1686dbe93c42b597f32422b6e8ac75 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:53.364679 24192 tablet_bootstrap.cc:492] T bdb710e59756492d870d1a152a629094 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:53.367352 24194 tablet_bootstrap.cc:492] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:53.370424 24195 tablet_bootstrap.cc:654] T 6f1686dbe93c42b597f32422b6e8ac75 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.370769 24192 tablet_bootstrap.cc:654] T bdb710e59756492d870d1a152a629094 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.371811 24194 tablet_bootstrap.cc:654] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.372084 24193 tablet_bootstrap.cc:492] T 6f1686dbe93c42b597f32422b6e8ac75 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:53.375757 24193 tablet_bootstrap.cc:654] T 6f1686dbe93c42b597f32422b6e8ac75 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.380079 24194 tablet_bootstrap.cc:492] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:53.380079 24195 tablet_bootstrap.cc:492] T 6f1686dbe93c42b597f32422b6e8ac75 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:53.380177 24193 tablet_bootstrap.cc:492] T 6f1686dbe93c42b597f32422b6e8ac75 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:53.380252 24195 ts_tablet_manager.cc:1367] T 6f1686dbe93c42b597f32422b6e8ac75 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.003s
I20240502 01:53:53.380252 24194 ts_tablet_manager.cc:1367] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20240502 01:53:53.380355 24193 ts_tablet_manager.cc:1367] T 6f1686dbe93c42b597f32422b6e8ac75 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 01:53:53.381055 24192 tablet_bootstrap.cc:492] T bdb710e59756492d870d1a152a629094 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:53.381243 24192 ts_tablet_manager.cc:1367] T bdb710e59756492d870d1a152a629094 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.005s
I20240502 01:53:53.381642 24194 raft_consensus.cc:348] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:53.381861 24194 raft_consensus.cc:374] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.381951 24194 raft_consensus.cc:729] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.382117 24192 raft_consensus.cc:348] T bdb710e59756492d870d1a152a629094 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:53.382329 24192 raft_consensus.cc:374] T bdb710e59756492d870d1a152a629094 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.382254 24194 consensus_queue.cc:260] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:53.382407 24192 raft_consensus.cc:729] T bdb710e59756492d870d1a152a629094 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.382572 24193 raft_consensus.cc:348] T 6f1686dbe93c42b597f32422b6e8ac75 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:53.382865 24193 raft_consensus.cc:374] T 6f1686dbe93c42b597f32422b6e8ac75 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.382997 24193 raft_consensus.cc:729] T 6f1686dbe93c42b597f32422b6e8ac75 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.382922 24192 consensus_queue.cc:260] T bdb710e59756492d870d1a152a629094 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:53.383399 24193 consensus_queue.cc:260] T 6f1686dbe93c42b597f32422b6e8ac75 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:53.383724 24192 ts_tablet_manager.cc:1398] T bdb710e59756492d870d1a152a629094 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:53.384027 24194 ts_tablet_manager.cc:1398] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:53.384106 24193 ts_tablet_manager.cc:1398] T 6f1686dbe93c42b597f32422b6e8ac75 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:53.384145 24192 tablet_bootstrap.cc:492] T b993ed799224421fa6fc29c4cf6dfbd0 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:53.384325 24194 tablet_bootstrap.cc:492] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:53.384474 24193 tablet_bootstrap.cc:492] T bdb710e59756492d870d1a152a629094 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:53.385936 24194 tablet_bootstrap.cc:654] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.386310 24193 tablet_bootstrap.cc:654] T bdb710e59756492d870d1a152a629094 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.390623 24192 tablet_bootstrap.cc:654] T b993ed799224421fa6fc29c4cf6dfbd0 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.391769 24195 raft_consensus.cc:348] T 6f1686dbe93c42b597f32422b6e8ac75 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:53.392050 24195 raft_consensus.cc:374] T 6f1686dbe93c42b597f32422b6e8ac75 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.392189 24195 raft_consensus.cc:729] T 6f1686dbe93c42b597f32422b6e8ac75 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.392453 24195 consensus_queue.cc:260] T 6f1686dbe93c42b597f32422b6e8ac75 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:53.394613 24195 ts_tablet_manager.cc:1398] T 6f1686dbe93c42b597f32422b6e8ac75 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240502 01:53:53.395016 24195 tablet_bootstrap.cc:492] T b993ed799224421fa6fc29c4cf6dfbd0 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:53.396788 24194 tablet_bootstrap.cc:492] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:53.397042 24194 ts_tablet_manager.cc:1367] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20240502 01:53:53.397928 24194 raft_consensus.cc:348] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:53.398409 24194 raft_consensus.cc:374] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.398496 24194 raft_consensus.cc:729] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.398758 24194 consensus_queue.cc:260] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:53.400799 24194 ts_tablet_manager.cc:1398] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:53.401252 24194 tablet_bootstrap.cc:492] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:53.401947 24195 tablet_bootstrap.cc:654] T b993ed799224421fa6fc29c4cf6dfbd0 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.403438 24194 tablet_bootstrap.cc:654] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.405915 24194 tablet_bootstrap.cc:492] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:53.406087 24194 ts_tablet_manager.cc:1367] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:53:53.407003 24194 raft_consensus.cc:348] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:53.407213 24194 raft_consensus.cc:374] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.407408 24194 raft_consensus.cc:729] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.407677 24194 consensus_queue.cc:260] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:53.408233 24194 ts_tablet_manager.cc:1398] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:53:53.409622 24192 tablet_bootstrap.cc:492] T b993ed799224421fa6fc29c4cf6dfbd0 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:53.409870 24192 ts_tablet_manager.cc:1367] T b993ed799224421fa6fc29c4cf6dfbd0 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.000s
I20240502 01:53:53.410593 24195 tablet_bootstrap.cc:492] T b993ed799224421fa6fc29c4cf6dfbd0 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:53.410776 24195 ts_tablet_manager.cc:1367] T b993ed799224421fa6fc29c4cf6dfbd0 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.003s
I20240502 01:53:53.411254 24192 raft_consensus.cc:348] T b993ed799224421fa6fc29c4cf6dfbd0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:53.411618 24192 raft_consensus.cc:374] T b993ed799224421fa6fc29c4cf6dfbd0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.411679 24195 raft_consensus.cc:348] T b993ed799224421fa6fc29c4cf6dfbd0 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:53.411823 24192 raft_consensus.cc:729] T b993ed799224421fa6fc29c4cf6dfbd0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.411969 24195 raft_consensus.cc:374] T b993ed799224421fa6fc29c4cf6dfbd0 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.412065 24195 raft_consensus.cc:729] T b993ed799224421fa6fc29c4cf6dfbd0 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.412220 24193 tablet_bootstrap.cc:492] T bdb710e59756492d870d1a152a629094 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:53.412393 24193 ts_tablet_manager.cc:1367] T bdb710e59756492d870d1a152a629094 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.000s
I20240502 01:53:53.412389 24195 consensus_queue.cc:260] T b993ed799224421fa6fc29c4cf6dfbd0 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:53.412691 24192 consensus_queue.cc:260] T b993ed799224421fa6fc29c4cf6dfbd0 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:53.413352 24192 ts_tablet_manager.cc:1398] T b993ed799224421fa6fc29c4cf6dfbd0 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:53.413396 24193 raft_consensus.cc:348] T bdb710e59756492d870d1a152a629094 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:53.413647 24193 raft_consensus.cc:374] T bdb710e59756492d870d1a152a629094 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.413727 24193 raft_consensus.cc:729] T bdb710e59756492d870d1a152a629094 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.413980 24193 consensus_queue.cc:260] T bdb710e59756492d870d1a152a629094 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:53.414119 24195 ts_tablet_manager.cc:1398] T b993ed799224421fa6fc29c4cf6dfbd0 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 01:53:53.414721 24193 ts_tablet_manager.cc:1398] T bdb710e59756492d870d1a152a629094 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:53.431465 23927 raft_consensus.cc:482] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:53.431670 23927 raft_consensus.cc:504] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:53.432601 23927 leader_election.cc:288] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:53.433255 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b993ed799224421fa6fc29c4cf6dfbd0" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:53.433279 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b993ed799224421fa6fc29c4cf6dfbd0" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:53.433506 22249 raft_consensus.cc:2454] T b993ed799224421fa6fc29c4cf6dfbd0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:53.433599 22119 raft_consensus.cc:2454] T b993ed799224421fa6fc29c4cf6dfbd0 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:53.434032 21924 leader_election.cc:302] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:53.434332 23927 raft_consensus.cc:2781] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:53.434461 23927 raft_consensus.cc:482] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:53.434574 23927 raft_consensus.cc:3037] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.436549 23927 raft_consensus.cc:504] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:53.437109 23927 leader_election.cc:288] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:53.437652 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b993ed799224421fa6fc29c4cf6dfbd0" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:53.437681 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b993ed799224421fa6fc29c4cf6dfbd0" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:53.437875 22249 raft_consensus.cc:3037] T b993ed799224421fa6fc29c4cf6dfbd0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.437906 22119 raft_consensus.cc:3037] T b993ed799224421fa6fc29c4cf6dfbd0 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.440795 22119 raft_consensus.cc:2454] T b993ed799224421fa6fc29c4cf6dfbd0 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:53.440874 22249 raft_consensus.cc:2454] T b993ed799224421fa6fc29c4cf6dfbd0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:53.441402 21924 leader_election.cc:302] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:53.441696 23927 raft_consensus.cc:2781] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:53.442173 23927 raft_consensus.cc:686] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:53:53.442775 23927 consensus_queue.cc:237] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:53.446254 21712 catalog_manager.cc:5484] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:53:53.651396 23927 raft_consensus.cc:482] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:53.651692 23927 raft_consensus.cc:504] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:53.652571 23927 leader_election.cc:288] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:53.653048 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb710e59756492d870d1a152a629094" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:53.653179 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb710e59756492d870d1a152a629094" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:53.653349 21857 raft_consensus.cc:2454] T bdb710e59756492d870d1a152a629094 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:53.653435 22249 raft_consensus.cc:2454] T bdb710e59756492d870d1a152a629094 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:53.653856 21924 leader_election.cc:302] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:53.654141 23927 raft_consensus.cc:2781] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:53.654224 23927 raft_consensus.cc:482] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:53.654338 23927 raft_consensus.cc:3037] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.656297 23927 raft_consensus.cc:504] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:53.656908 23927 leader_election.cc:288] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:53.657310 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb710e59756492d870d1a152a629094" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:53.657476 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb710e59756492d870d1a152a629094" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:53.657549 21857 raft_consensus.cc:3037] T bdb710e59756492d870d1a152a629094 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.657681 22249 raft_consensus.cc:3037] T bdb710e59756492d870d1a152a629094 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.660043 22249 raft_consensus.cc:2454] T bdb710e59756492d870d1a152a629094 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:53.660089 21857 raft_consensus.cc:2454] T bdb710e59756492d870d1a152a629094 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:53.660480 21924 leader_election.cc:302] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:53.660815 23927 raft_consensus.cc:2781] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:53.661000 23927 raft_consensus.cc:686] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:53:53.661271 23927 consensus_queue.cc:237] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:53.664803 21714 catalog_manager.cc:5484] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:53:53.804874 23927 raft_consensus.cc:482] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:53.805112 23927 raft_consensus.cc:504] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:53.805968 23927 leader_election.cc:288] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:53.806244 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1686dbe93c42b597f32422b6e8ac75" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:53.806512 22118 raft_consensus.cc:2454] T 6f1686dbe93c42b597f32422b6e8ac75 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:53.806587 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1686dbe93c42b597f32422b6e8ac75" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:53.806790 21857 raft_consensus.cc:2454] T 6f1686dbe93c42b597f32422b6e8ac75 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:53.806978 21924 leader_election.cc:302] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:53.807276 23927 raft_consensus.cc:2781] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:53.807425 23927 raft_consensus.cc:482] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:53.807540 23927 raft_consensus.cc:3037] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.810271 23927 raft_consensus.cc:504] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:53.810954 23927 leader_election.cc:288] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:53.811311 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1686dbe93c42b597f32422b6e8ac75" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:53.811548 22119 raft_consensus.cc:3037] T 6f1686dbe93c42b597f32422b6e8ac75 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.811501 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1686dbe93c42b597f32422b6e8ac75" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:53.811704 21857 raft_consensus.cc:3037] T 6f1686dbe93c42b597f32422b6e8ac75 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.813688 22119 raft_consensus.cc:2454] T 6f1686dbe93c42b597f32422b6e8ac75 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:53.813743 21857 raft_consensus.cc:2454] T 6f1686dbe93c42b597f32422b6e8ac75 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:53.814160 21924 leader_election.cc:302] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:53.814469 23927 raft_consensus.cc:2781] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:53.814635 23927 raft_consensus.cc:686] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:53:53.814918 23927 consensus_queue.cc:237] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:53.818055 21714 catalog_manager.cc:5484] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:53:53.852474 23927 consensus_queue.cc:1035] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:53.874400 23927 consensus_queue.cc:1035] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:54.200556 24210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:54.200945 24210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:54.208896 24210 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240502 01:53:54.265458 24213 consensus_queue.cc:1035] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:54.272277 24213 consensus_queue.cc:1035] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:54.301887 24213 consensus_queue.cc:1035] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:54.322332 24213 consensus_queue.cc:1035] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 335.918 ms
time per row: 0.167959 ms
I20240502 01:53:54.724264 21713 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:54.734346 21837 tablet_service.cc:1467] Processing CreateTablet for tablet d52f4b5c815f4188a09eb12ec5375361 (DEFAULT_TABLE table=rebalance_test_table_17 [id=7a17c6db79004eb696e7dd281fdc4f3b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:54.734846 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52f4b5c815f4188a09eb12ec5375361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.734807 22098 tablet_service.cc:1467] Processing CreateTablet for tablet d52f4b5c815f4188a09eb12ec5375361 (DEFAULT_TABLE table=rebalance_test_table_17 [id=7a17c6db79004eb696e7dd281fdc4f3b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:54.735263 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52f4b5c815f4188a09eb12ec5375361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.735353 22231 tablet_service.cc:1467] Processing CreateTablet for tablet d52f4b5c815f4188a09eb12ec5375361 (DEFAULT_TABLE table=rebalance_test_table_17 [id=7a17c6db79004eb696e7dd281fdc4f3b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:54.735459 22230 tablet_service.cc:1467] Processing CreateTablet for tablet ad3cd6fff977431b9cd5df6d083bef80 (DEFAULT_TABLE table=rebalance_test_table_17 [id=7a17c6db79004eb696e7dd281fdc4f3b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:54.735803 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52f4b5c815f4188a09eb12ec5375361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.736560 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3cd6fff977431b9cd5df6d083bef80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.738435 21835 tablet_service.cc:1467] Processing CreateTablet for tablet ad3cd6fff977431b9cd5df6d083bef80 (DEFAULT_TABLE table=rebalance_test_table_17 [id=7a17c6db79004eb696e7dd281fdc4f3b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:54.738479 22099 tablet_service.cc:1467] Processing CreateTablet for tablet ad3cd6fff977431b9cd5df6d083bef80 (DEFAULT_TABLE table=rebalance_test_table_17 [id=7a17c6db79004eb696e7dd281fdc4f3b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:54.738826 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3cd6fff977431b9cd5df6d083bef80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.738868 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3cd6fff977431b9cd5df6d083bef80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.741333 22229 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8200b044ef44e1a0451baf2b1485d2 (DEFAULT_TABLE table=rebalance_test_table_17 [id=7a17c6db79004eb696e7dd281fdc4f3b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:54.741663 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8200b044ef44e1a0451baf2b1485d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.745033 21968 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8200b044ef44e1a0451baf2b1485d2 (DEFAULT_TABLE table=rebalance_test_table_17 [id=7a17c6db79004eb696e7dd281fdc4f3b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:54.745509 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8200b044ef44e1a0451baf2b1485d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.745851 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8200b044ef44e1a0451baf2b1485d2 (DEFAULT_TABLE table=rebalance_test_table_17 [id=7a17c6db79004eb696e7dd281fdc4f3b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:54.746342 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8200b044ef44e1a0451baf2b1485d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.757490 24276 tablet_bootstrap.cc:492] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:54.761983 24276 tablet_bootstrap.cc:654] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:54.766198 24275 tablet_bootstrap.cc:492] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:54.767235 24277 tablet_bootstrap.cc:492] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:54.769137 24277 tablet_bootstrap.cc:654] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:54.771116 24275 tablet_bootstrap.cc:654] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:54.771291 24277 tablet_bootstrap.cc:492] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:54.771426 24276 tablet_bootstrap.cc:492] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:54.771483 24277 ts_tablet_manager.cc:1367] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240502 01:53:54.771641 24276 ts_tablet_manager.cc:1367] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20240502 01:53:54.772449 24277 raft_consensus.cc:348] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:54.772774 24277 raft_consensus.cc:374] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:54.772958 24277 raft_consensus.cc:729] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:54.773314 24277 consensus_queue.cc:260] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:54.773530 24274 tablet_bootstrap.cc:492] T d52f4b5c815f4188a09eb12ec5375361 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:54.773538 24276 raft_consensus.cc:348] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:54.773787 24276 raft_consensus.cc:374] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:54.773903 24276 raft_consensus.cc:729] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:54.774132 24277 ts_tablet_manager.cc:1398] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:54.774194 24276 consensus_queue.cc:260] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:54.774796 24276 ts_tablet_manager.cc:1398] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:53:54.775101 24276 tablet_bootstrap.cc:492] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:54.775367 24274 tablet_bootstrap.cc:654] T d52f4b5c815f4188a09eb12ec5375361 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:54.776971 24276 tablet_bootstrap.cc:654] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:54.782666 24274 tablet_bootstrap.cc:492] T d52f4b5c815f4188a09eb12ec5375361 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:54.782832 24274 ts_tablet_manager.cc:1367] T d52f4b5c815f4188a09eb12ec5375361 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20240502 01:53:54.783538 24274 raft_consensus.cc:348] T d52f4b5c815f4188a09eb12ec5375361 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:54.783713 24274 raft_consensus.cc:374] T d52f4b5c815f4188a09eb12ec5375361 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:54.785002 24274 raft_consensus.cc:729] T d52f4b5c815f4188a09eb12ec5375361 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:54.785494 24274 consensus_queue.cc:260] T d52f4b5c815f4188a09eb12ec5375361 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:54.786393 24274 ts_tablet_manager.cc:1398] T d52f4b5c815f4188a09eb12ec5375361 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240502 01:53:54.794222 24274 tablet_bootstrap.cc:492] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:54.796550 24274 tablet_bootstrap.cc:654] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:54.805836 24274 tablet_bootstrap.cc:492] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:54.806152 24274 ts_tablet_manager.cc:1367] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20240502 01:53:54.806521 24275 tablet_bootstrap.cc:492] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:54.806612 24276 tablet_bootstrap.cc:492] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:54.806701 24275 ts_tablet_manager.cc:1367] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.013s
I20240502 01:53:54.806933 24276 ts_tablet_manager.cc:1367] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.015s
I20240502 01:53:54.807133 24274 raft_consensus.cc:348] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:54.807456 24274 raft_consensus.cc:374] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:54.807583 24274 raft_consensus.cc:729] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:54.807858 24276 raft_consensus.cc:348] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:54.807875 24275 raft_consensus.cc:348] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:54.808089 24275 raft_consensus.cc:374] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:54.808023 24274 consensus_queue.cc:260] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:54.808193 24275 raft_consensus.cc:729] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:54.808230 24276 raft_consensus.cc:374] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:54.808326 24276 raft_consensus.cc:729] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:54.808660 24276 consensus_queue.cc:260] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:54.809860 24274 ts_tablet_manager.cc:1398] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:53:54.810801 24276 ts_tablet_manager.cc:1398] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:53:54.808501 24275 consensus_queue.cc:260] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:54.812382 24275 ts_tablet_manager.cc:1398] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 01:53:54.812781 24275 tablet_bootstrap.cc:492] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:54.814697 24275 tablet_bootstrap.cc:654] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:54.817729 24276 tablet_bootstrap.cc:492] T ad3cd6fff977431b9cd5df6d083bef80 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:54.820092 24276 tablet_bootstrap.cc:654] T ad3cd6fff977431b9cd5df6d083bef80 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:54.822693 24275 tablet_bootstrap.cc:492] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:54.823016 24275 ts_tablet_manager.cc:1367] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.003s
I20240502 01:53:54.824659 24275 raft_consensus.cc:348] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:54.825089 24275 raft_consensus.cc:374] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:54.825248 24275 raft_consensus.cc:729] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:54.825704 24275 consensus_queue.cc:260] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:54.826586 24275 ts_tablet_manager.cc:1398] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:54.827005 24275 tablet_bootstrap.cc:492] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:54.829572 24275 tablet_bootstrap.cc:654] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:54.830461 24276 tablet_bootstrap.cc:492] T ad3cd6fff977431b9cd5df6d083bef80 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:54.830615 24276 ts_tablet_manager.cc:1367] T ad3cd6fff977431b9cd5df6d083bef80 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20240502 01:53:54.831195 24276 raft_consensus.cc:348] T ad3cd6fff977431b9cd5df6d083bef80 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:54.831398 24276 raft_consensus.cc:374] T ad3cd6fff977431b9cd5df6d083bef80 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:54.831519 24276 raft_consensus.cc:729] T ad3cd6fff977431b9cd5df6d083bef80 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:54.831808 24276 consensus_queue.cc:260] T ad3cd6fff977431b9cd5df6d083bef80 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:54.832579 24276 ts_tablet_manager.cc:1398] T ad3cd6fff977431b9cd5df6d083bef80 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:54.846006 24275 tablet_bootstrap.cc:492] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:54.846254 24275 ts_tablet_manager.cc:1367] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.008s
I20240502 01:53:54.847157 24275 raft_consensus.cc:348] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:54.847429 24275 raft_consensus.cc:374] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:54.847561 24275 raft_consensus.cc:729] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:54.847859 24275 consensus_queue.cc:260] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:54.848490 24275 ts_tablet_manager.cc:1398] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:54.849910 24205 raft_consensus.cc:482] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:54.850080 24205 raft_consensus.cc:504] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:54.851092 24205 leader_election.cc:288] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:54.851573 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8200b044ef44e1a0451baf2b1485d2" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:54.851827 22250 raft_consensus.cc:2454] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:54.851960 21855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8200b044ef44e1a0451baf2b1485d2" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:54.852183 21855 raft_consensus.cc:2454] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:54.852311 21924 leader_election.cc:302] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:54.853296 24213 raft_consensus.cc:2781] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:54.853401 24213 raft_consensus.cc:482] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:54.853502 24213 raft_consensus.cc:3037] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:54.856412 24213 raft_consensus.cc:504] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:54.857643 24213 leader_election.cc:288] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:54.858073 21855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8200b044ef44e1a0451baf2b1485d2" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:54.858283 21855 raft_consensus.cc:3037] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:54.858273 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8200b044ef44e1a0451baf2b1485d2" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:54.858506 22250 raft_consensus.cc:3037] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:54.860576 21855 raft_consensus.cc:2454] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:54.861090 21922 leader_election.cc:302] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:53:54.861450 22250 raft_consensus.cc:2454] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:54.862423 24213 raft_consensus.cc:2781] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:54.862721 24213 raft_consensus.cc:686] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:53:54.863390 24213 consensus_queue.cc:237] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:54.875341 21714 catalog_manager.cc:5484] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:53:54.985298 23967 raft_consensus.cc:482] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:54.985508 23967 raft_consensus.cc:504] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:54.986212 23967 leader_election.cc:288] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:54.986548 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3cd6fff977431b9cd5df6d083bef80" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:54.986696 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3cd6fff977431b9cd5df6d083bef80" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:54.986837 21857 raft_consensus.cc:2454] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:54.986944 22250 raft_consensus.cc:2454] T ad3cd6fff977431b9cd5df6d083bef80 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:54.987284 22053 leader_election.cc:302] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:54.987565 23967 raft_consensus.cc:2781] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:54.987680 23967 raft_consensus.cc:482] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:54.987759 23967 raft_consensus.cc:3037] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:54.989704 23967 raft_consensus.cc:504] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:54.990247 23967 leader_election.cc:288] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:54.990672 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3cd6fff977431b9cd5df6d083bef80" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:54.990783 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3cd6fff977431b9cd5df6d083bef80" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:54.990912 21857 raft_consensus.cc:3037] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:54.990998 22250 raft_consensus.cc:3037] T ad3cd6fff977431b9cd5df6d083bef80 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:54.993798 21857 raft_consensus.cc:2454] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:54.993815 22250 raft_consensus.cc:2454] T ad3cd6fff977431b9cd5df6d083bef80 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:54.994242 22053 leader_election.cc:302] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:54.994537 23967 raft_consensus.cc:2781] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:54.994673 23967 raft_consensus.cc:686] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:53:54.994948 23967 consensus_queue.cc:237] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:54.998605 21714 catalog_manager.cc:5484] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:53:55.170292 24259 raft_consensus.cc:482] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:55.170511 24259 raft_consensus.cc:504] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:55.171239 24259 leader_election.cc:288] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:55.171583 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52f4b5c815f4188a09eb12ec5375361" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:55.171830 22250 raft_consensus.cc:2454] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:55.171953 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52f4b5c815f4188a09eb12ec5375361" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:55.172201 22119 raft_consensus.cc:2454] T d52f4b5c815f4188a09eb12ec5375361 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:55.172245 21792 leader_election.cc:302] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:55.172520 24259 raft_consensus.cc:2781] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:55.172659 24259 raft_consensus.cc:482] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:55.172756 24259 raft_consensus.cc:3037] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:55.174626 24259 raft_consensus.cc:504] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:55.175143 24259 leader_election.cc:288] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:55.175577 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52f4b5c815f4188a09eb12ec5375361" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:55.175621 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52f4b5c815f4188a09eb12ec5375361" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:55.175788 22250 raft_consensus.cc:3037] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:55.175817 22119 raft_consensus.cc:3037] T d52f4b5c815f4188a09eb12ec5375361 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:55.177621 22250 raft_consensus.cc:2454] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:55.177703 22119 raft_consensus.cc:2454] T d52f4b5c815f4188a09eb12ec5375361 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:55.178045 21792 leader_election.cc:302] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:55.178335 24259 raft_consensus.cc:2781] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:55.178503 24259 raft_consensus.cc:686] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:53:55.179121 24259 consensus_queue.cc:237] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:55.182755 21714 catalog_manager.cc:5484] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:53:55.269326 24213 consensus_queue.cc:1035] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:55.280468 24213 consensus_queue.cc:1035] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:55.446307 23967 consensus_queue.cc:1035] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:55.453181 24299 consensus_queue.cc:1035] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:55.613926 24290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:55.614308 24290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:55.623463 24290 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240502 01:53:55.672561 24259 consensus_queue.cc:1035] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:55.677512 24259 consensus_queue.cc:1035] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 284.306 ms
time per row: 0.142153 ms
I20240502 01:53:56.070977 21709 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:56.080421 21968 tablet_service.cc:1467] Processing CreateTablet for tablet 071ad932ce95499fac4d7af7292b42ff (DEFAULT_TABLE table=rebalance_test_table_18 [id=596028c83e784a669c0b5819444e54be]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:56.080986 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071ad932ce95499fac4d7af7292b42ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.081080 21835 tablet_service.cc:1467] Processing CreateTablet for tablet 071ad932ce95499fac4d7af7292b42ff (DEFAULT_TABLE table=rebalance_test_table_18 [id=596028c83e784a669c0b5819444e54be]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:56.081379 22099 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb367ab8254423c93c7f33f885d374f (DEFAULT_TABLE table=rebalance_test_table_18 [id=596028c83e784a669c0b5819444e54be]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:56.081563 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071ad932ce95499fac4d7af7292b42ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.081810 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eb367ab8254423c93c7f33f885d374f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.082135 21967 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb367ab8254423c93c7f33f885d374f (DEFAULT_TABLE table=rebalance_test_table_18 [id=596028c83e784a669c0b5819444e54be]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:56.082532 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eb367ab8254423c93c7f33f885d374f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.081393 22098 tablet_service.cc:1467] Processing CreateTablet for tablet 071ad932ce95499fac4d7af7292b42ff (DEFAULT_TABLE table=rebalance_test_table_18 [id=596028c83e784a669c0b5819444e54be]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:56.087221 22229 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb367ab8254423c93c7f33f885d374f (DEFAULT_TABLE table=rebalance_test_table_18 [id=596028c83e784a669c0b5819444e54be]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:56.087440 21969 tablet_service.cc:1467] Processing CreateTablet for tablet 37e185badcf64f078e0df67837cfe054 (DEFAULT_TABLE table=rebalance_test_table_18 [id=596028c83e784a669c0b5819444e54be]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:56.087661 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eb367ab8254423c93c7f33f885d374f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.087911 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e185badcf64f078e0df67837cfe054. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.093137 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071ad932ce95499fac4d7af7292b42ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.093981 21837 tablet_service.cc:1467] Processing CreateTablet for tablet 37e185badcf64f078e0df67837cfe054 (DEFAULT_TABLE table=rebalance_test_table_18 [id=596028c83e784a669c0b5819444e54be]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:56.094460 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e185badcf64f078e0df67837cfe054. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.098306 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 37e185badcf64f078e0df67837cfe054 (DEFAULT_TABLE table=rebalance_test_table_18 [id=596028c83e784a669c0b5819444e54be]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:56.098732 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e185badcf64f078e0df67837cfe054. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.104640 24357 tablet_bootstrap.cc:492] T 5eb367ab8254423c93c7f33f885d374f P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:56.106582 24357 tablet_bootstrap.cc:654] T 5eb367ab8254423c93c7f33f885d374f P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:56.110376 24356 tablet_bootstrap.cc:492] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:56.112380 24356 tablet_bootstrap.cc:654] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:56.112679 24357 tablet_bootstrap.cc:492] T 5eb367ab8254423c93c7f33f885d374f P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:56.112839 24357 ts_tablet_manager.cc:1367] T 5eb367ab8254423c93c7f33f885d374f P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240502 01:53:56.113721 24357 raft_consensus.cc:348] T 5eb367ab8254423c93c7f33f885d374f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:56.114100 24357 raft_consensus.cc:374] T 5eb367ab8254423c93c7f33f885d374f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:56.114246 24357 raft_consensus.cc:729] T 5eb367ab8254423c93c7f33f885d374f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:56.114630 24357 consensus_queue.cc:260] T 5eb367ab8254423c93c7f33f885d374f P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:56.115386 24357 ts_tablet_manager.cc:1398] T 5eb367ab8254423c93c7f33f885d374f P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:56.117605 24356 tablet_bootstrap.cc:492] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:56.117816 24356 ts_tablet_manager.cc:1367] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240502 01:53:56.118979 24356 raft_consensus.cc:348] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:56.119249 24356 raft_consensus.cc:374] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:56.119390 24356 raft_consensus.cc:729] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:56.119747 24356 consensus_queue.cc:260] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:56.125535 24356 ts_tablet_manager.cc:1398] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.008s user 0.002s sys 0.002s
I20240502 01:53:56.126531 24356 tablet_bootstrap.cc:492] T 37e185badcf64f078e0df67837cfe054 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:56.128937 24356 tablet_bootstrap.cc:654] T 37e185badcf64f078e0df67837cfe054 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:56.131834 24359 tablet_bootstrap.cc:492] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:56.133574 24355 tablet_bootstrap.cc:492] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:56.133806 24359 tablet_bootstrap.cc:654] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:56.135879 24355 tablet_bootstrap.cc:654] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:56.136234 24359 tablet_bootstrap.cc:492] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:56.136438 24359 ts_tablet_manager.cc:1367] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:53:56.137399 24359 raft_consensus.cc:348] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:56.137866 24359 raft_consensus.cc:374] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:56.138137 24359 raft_consensus.cc:729] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:56.138531 24359 consensus_queue.cc:260] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:56.140597 24359 ts_tablet_manager.cc:1398] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:56.140976 24359 tablet_bootstrap.cc:492] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:56.140992 24356 tablet_bootstrap.cc:492] T 37e185badcf64f078e0df67837cfe054 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:56.141184 24356 ts_tablet_manager.cc:1367] T 37e185badcf64f078e0df67837cfe054 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.010s
I20240502 01:53:56.141819 24356 raft_consensus.cc:348] T 37e185badcf64f078e0df67837cfe054 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:56.142011 24356 raft_consensus.cc:374] T 37e185badcf64f078e0df67837cfe054 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:56.142097 24356 raft_consensus.cc:729] T 37e185badcf64f078e0df67837cfe054 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:56.142342 24356 consensus_queue.cc:260] T 37e185badcf64f078e0df67837cfe054 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:56.142884 24359 tablet_bootstrap.cc:654] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:56.143014 24356 ts_tablet_manager.cc:1398] T 37e185badcf64f078e0df67837cfe054 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:56.145004 24359 tablet_bootstrap.cc:492] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:56.145242 24359 ts_tablet_manager.cc:1367] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 01:53:56.146268 24359 raft_consensus.cc:348] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:56.146615 24359 raft_consensus.cc:374] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:56.146761 24359 raft_consensus.cc:729] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:56.147091 24359 consensus_queue.cc:260] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:56.147938 24359 ts_tablet_manager.cc:1398] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:56.148273 24359 tablet_bootstrap.cc:492] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:56.150002 24359 tablet_bootstrap.cc:654] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:56.151078 24355 tablet_bootstrap.cc:492] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:56.151271 24355 ts_tablet_manager.cc:1367] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20240502 01:53:56.152154 24355 raft_consensus.cc:348] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:56.152395 24355 raft_consensus.cc:374] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:56.152506 24355 raft_consensus.cc:729] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:56.152740 24355 consensus_queue.cc:260] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:56.153378 24355 ts_tablet_manager.cc:1398] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:56.153857 24355 tablet_bootstrap.cc:492] T 5eb367ab8254423c93c7f33f885d374f P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:56.157088 24359 tablet_bootstrap.cc:492] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:56.157265 24359 ts_tablet_manager.cc:1367] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20240502 01:53:56.157898 24355 tablet_bootstrap.cc:654] T 5eb367ab8254423c93c7f33f885d374f P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:56.158236 24359 raft_consensus.cc:348] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:56.158463 24359 raft_consensus.cc:374] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:56.158597 24359 raft_consensus.cc:729] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:56.158937 24359 consensus_queue.cc:260] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:56.159691 24355 tablet_bootstrap.cc:492] T 5eb367ab8254423c93c7f33f885d374f P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:56.159857 24359 ts_tablet_manager.cc:1398] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:56.159862 24355 ts_tablet_manager.cc:1367] T 5eb367ab8254423c93c7f33f885d374f P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 01:53:56.160606 24355 raft_consensus.cc:348] T 5eb367ab8254423c93c7f33f885d374f P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:56.160840 24355 raft_consensus.cc:374] T 5eb367ab8254423c93c7f33f885d374f P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:56.160951 24355 raft_consensus.cc:729] T 5eb367ab8254423c93c7f33f885d374f P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:56.161209 24355 consensus_queue.cc:260] T 5eb367ab8254423c93c7f33f885d374f P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:56.161859 24355 ts_tablet_manager.cc:1398] T 5eb367ab8254423c93c7f33f885d374f P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:56.162381 24355 tablet_bootstrap.cc:492] T 37e185badcf64f078e0df67837cfe054 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:56.163970 24355 tablet_bootstrap.cc:654] T 37e185badcf64f078e0df67837cfe054 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:56.167033 24355 tablet_bootstrap.cc:492] T 37e185badcf64f078e0df67837cfe054 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:56.167212 24355 ts_tablet_manager.cc:1367] T 37e185badcf64f078e0df67837cfe054 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20240502 01:53:56.168048 24355 raft_consensus.cc:348] T 37e185badcf64f078e0df67837cfe054 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:56.168272 24355 raft_consensus.cc:374] T 37e185badcf64f078e0df67837cfe054 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:56.168393 24355 raft_consensus.cc:729] T 37e185badcf64f078e0df67837cfe054 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:56.168658 24355 consensus_queue.cc:260] T 37e185badcf64f078e0df67837cfe054 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:56.169232 24355 ts_tablet_manager.cc:1398] T 37e185badcf64f078e0df67837cfe054 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:56.328989 24299 raft_consensus.cc:482] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:56.329262 24299 raft_consensus.cc:504] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:56.330129 24299 leader_election.cc:288] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:56.330618 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb367ab8254423c93c7f33f885d374f" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:56.330868 21986 raft_consensus.cc:2454] T 5eb367ab8254423c93c7f33f885d374f P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:56.330962 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb367ab8254423c93c7f33f885d374f" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:56.331238 22249 raft_consensus.cc:2454] T 5eb367ab8254423c93c7f33f885d374f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:56.331763 22053 leader_election.cc:302] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:56.332093 24299 raft_consensus.cc:2781] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:56.332211 24299 raft_consensus.cc:482] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:56.332329 24299 raft_consensus.cc:3037] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:56.334898 24299 raft_consensus.cc:504] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:56.335918 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb367ab8254423c93c7f33f885d374f" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:56.335994 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb367ab8254423c93c7f33f885d374f" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:56.336146 21986 raft_consensus.cc:3037] T 5eb367ab8254423c93c7f33f885d374f P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:56.336162 22249 raft_consensus.cc:3037] T 5eb367ab8254423c93c7f33f885d374f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:56.338866 24299 leader_election.cc:288] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:56.339068 21986 raft_consensus.cc:2454] T 5eb367ab8254423c93c7f33f885d374f P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:56.339382 22249 raft_consensus.cc:2454] T 5eb367ab8254423c93c7f33f885d374f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:56.339555 22053 leader_election.cc:302] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:56.339962 24299 raft_consensus.cc:2781] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:56.340167 24299 raft_consensus.cc:686] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:53:56.340472 24299 consensus_queue.cc:237] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:53:56.344199 21709 catalog_manager.cc:5484] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:53:56.348553 24299 raft_consensus.cc:482] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:56.348696 24299 raft_consensus.cc:504] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:56.349260 24299 leader_election.cc:288] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:56.349823 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e185badcf64f078e0df67837cfe054" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:56.349864 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e185badcf64f078e0df67837cfe054" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:56.350065 21986 raft_consensus.cc:2454] T 37e185badcf64f078e0df67837cfe054 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:56.350133 21857 raft_consensus.cc:2454] T 37e185badcf64f078e0df67837cfe054 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:53:56.350591 22053 leader_election.cc:302] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:56.350886 24299 raft_consensus.cc:2781] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:56.351023 24299 raft_consensus.cc:482] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:56.351130 24299 raft_consensus.cc:3037] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:56.353540 24299 raft_consensus.cc:504] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:56.354029 24299 leader_election.cc:288] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:53:56.354418 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e185badcf64f078e0df67837cfe054" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:56.354511 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e185badcf64f078e0df67837cfe054" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:56.354630 21986 raft_consensus.cc:3037] T 37e185badcf64f078e0df67837cfe054 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:56.354739 21857 raft_consensus.cc:3037] T 37e185badcf64f078e0df67837cfe054 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:56.356547 21986 raft_consensus.cc:2454] T 37e185badcf64f078e0df67837cfe054 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:56.356701 21857 raft_consensus.cc:2454] T 37e185badcf64f078e0df67837cfe054 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:53:56.356969 22053 leader_election.cc:302] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:56.357298 24299 raft_consensus.cc:2781] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:56.357441 24299 raft_consensus.cc:686] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:53:56.357745 24299 consensus_queue.cc:237] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:56.361091 21709 catalog_manager.cc:5484] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:53:56.402248 24259 raft_consensus.cc:482] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:56.402487 24259 raft_consensus.cc:504] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:56.403260 24259 leader_election.cc:288] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:56.403681 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071ad932ce95499fac4d7af7292b42ff" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:56.403785 22117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071ad932ce95499fac4d7af7292b42ff" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:56.403975 21986 raft_consensus.cc:2454] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:56.404094 22117 raft_consensus.cc:2454] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:56.404404 21790 leader_election.cc:302] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:53:56.404778 24259 raft_consensus.cc:2781] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:56.404872 24259 raft_consensus.cc:482] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:56.405011 24259 raft_consensus.cc:3037] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:56.407186 24259 raft_consensus.cc:504] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:56.407781 24259 leader_election.cc:288] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:56.408300 22117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071ad932ce95499fac4d7af7292b42ff" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:56.408397 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071ad932ce95499fac4d7af7292b42ff" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:56.408524 22117 raft_consensus.cc:3037] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:56.408654 21988 raft_consensus.cc:3037] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:56.410815 21988 raft_consensus.cc:2454] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:56.410887 22117 raft_consensus.cc:2454] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:56.411370 21792 leader_election.cc:302] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:53:56.411712 24259 raft_consensus.cc:2781] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:56.411962 24259 raft_consensus.cc:686] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:53:56.412309 24259 consensus_queue.cc:237] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:56.416590 21709 catalog_manager.cc:5484] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:53:56.821062 24186 consensus_queue.cc:1035] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:56.830760 23967 consensus_queue.cc:1035] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:56.838537 24259 consensus_queue.cc:1035] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:56.842273 24186 consensus_queue.cc:1035] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:56.846572 23967 consensus_queue.cc:1035] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:56.869395 24259 consensus_queue.cc:1035] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:56.884932 24371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:56.885636 24371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:56.898128 24371 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
INSERT report
rows total: 2000
time total: 347.115 ms
time per row: 0.173557 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:53:58.438941 21706 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:58.450177 22229 tablet_service.cc:1467] Processing CreateTablet for tablet 84d602fa045e4cfa9b698d3e92e5cf8f (DEFAULT_TABLE table=rebalance_test_table_19 [id=9827c9aba79e4ba094c6e9c0e32f36f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:58.450423 21968 tablet_service.cc:1467] Processing CreateTablet for tablet 84d602fa045e4cfa9b698d3e92e5cf8f (DEFAULT_TABLE table=rebalance_test_table_19 [id=9827c9aba79e4ba094c6e9c0e32f36f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:58.450707 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d602fa045e4cfa9b698d3e92e5cf8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.450889 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d602fa045e4cfa9b698d3e92e5cf8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.450845 22230 tablet_service.cc:1467] Processing CreateTablet for tablet d4b6dfe0e8da4352b136d6f97d473bad (DEFAULT_TABLE table=rebalance_test_table_19 [id=9827c9aba79e4ba094c6e9c0e32f36f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:58.451218 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b6dfe0e8da4352b136d6f97d473bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.455814 24441 tablet_bootstrap.cc:492] T 84d602fa045e4cfa9b698d3e92e5cf8f P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:58.457463 21837 tablet_service.cc:1467] Processing CreateTablet for tablet 84d602fa045e4cfa9b698d3e92e5cf8f (DEFAULT_TABLE table=rebalance_test_table_19 [id=9827c9aba79e4ba094c6e9c0e32f36f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:58.457909 24441 tablet_bootstrap.cc:654] T 84d602fa045e4cfa9b698d3e92e5cf8f P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:58.457949 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d602fa045e4cfa9b698d3e92e5cf8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.458704 21969 tablet_service.cc:1467] Processing CreateTablet for tablet d4b6dfe0e8da4352b136d6f97d473bad (DEFAULT_TABLE table=rebalance_test_table_19 [id=9827c9aba79e4ba094c6e9c0e32f36f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:58.459110 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b6dfe0e8da4352b136d6f97d473bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.459524 22098 tablet_service.cc:1467] Processing CreateTablet for tablet 3a2f4ea104d4409383e5eb32ca2f78b0 (DEFAULT_TABLE table=rebalance_test_table_19 [id=9827c9aba79e4ba094c6e9c0e32f36f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:58.459961 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a2f4ea104d4409383e5eb32ca2f78b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.461277 22100 tablet_service.cc:1467] Processing CreateTablet for tablet d4b6dfe0e8da4352b136d6f97d473bad (DEFAULT_TABLE table=rebalance_test_table_19 [id=9827c9aba79e4ba094c6e9c0e32f36f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:58.461306 21835 tablet_service.cc:1467] Processing CreateTablet for tablet 3a2f4ea104d4409383e5eb32ca2f78b0 (DEFAULT_TABLE table=rebalance_test_table_19 [id=9827c9aba79e4ba094c6e9c0e32f36f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:58.461709 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b6dfe0e8da4352b136d6f97d473bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.461704 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a2f4ea104d4409383e5eb32ca2f78b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.471462 22230 tablet_service.cc:1467] Processing CreateTablet for tablet 3a2f4ea104d4409383e5eb32ca2f78b0 (DEFAULT_TABLE table=rebalance_test_table_19 [id=9827c9aba79e4ba094c6e9c0e32f36f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:58.472347 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a2f4ea104d4409383e5eb32ca2f78b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.476006 24445 tablet_bootstrap.cc:492] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:58.480103 24441 tablet_bootstrap.cc:492] T 84d602fa045e4cfa9b698d3e92e5cf8f P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:58.480185 24445 tablet_bootstrap.cc:654] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:58.481420 24441 ts_tablet_manager.cc:1367] T 84d602fa045e4cfa9b698d3e92e5cf8f P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240502 01:53:58.482995 24441 raft_consensus.cc:348] T 84d602fa045e4cfa9b698d3e92e5cf8f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:58.485489 24446 tablet_bootstrap.cc:492] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:58.485939 24441 raft_consensus.cc:374] T 84d602fa045e4cfa9b698d3e92e5cf8f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:58.486196 24441 raft_consensus.cc:729] T 84d602fa045e4cfa9b698d3e92e5cf8f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:58.486653 24441 consensus_queue.cc:260] T 84d602fa045e4cfa9b698d3e92e5cf8f P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:58.488485 24446 tablet_bootstrap.cc:654] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:58.489277 24441 ts_tablet_manager.cc:1398] T 84d602fa045e4cfa9b698d3e92e5cf8f P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:53:58.489578 24441 tablet_bootstrap.cc:492] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:58.490348 24445 tablet_bootstrap.cc:492] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:58.490562 24445 ts_tablet_manager.cc:1367] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20240502 01:53:58.490902 24441 tablet_bootstrap.cc:654] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:58.491408 24445 raft_consensus.cc:348] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:58.491745 24445 raft_consensus.cc:374] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:58.491863 24445 raft_consensus.cc:729] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:58.493899 24442 tablet_bootstrap.cc:492] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:58.495584 24442 tablet_bootstrap.cc:654] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:58.495630 24445 consensus_queue.cc:260] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:58.496084 23967 raft_consensus.cc:482] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:58.496250 23967 raft_consensus.cc:504] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:58.496515 24445 ts_tablet_manager.cc:1398] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240502 01:53:58.496863 24445 tablet_bootstrap.cc:492] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:53:58.497217 23967 leader_election.cc:288] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:53:58.497443 24441 tablet_bootstrap.cc:492] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:58.497598 24441 ts_tablet_manager.cc:1367] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:53:58.497694 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2f4ea104d4409383e5eb32ca2f78b0" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:58.498400 24441 raft_consensus.cc:348] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:58.498601 24441 raft_consensus.cc:374] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:58.498703 24441 raft_consensus.cc:729] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
W20240502 01:53:58.498674 22055 leader_election.cc:341] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:58.498819 24445 tablet_bootstrap.cc:654] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:58.498953 24441 consensus_queue.cc:260] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:58.499392 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2f4ea104d4409383e5eb32ca2f78b0" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:58.500320 24441 ts_tablet_manager.cc:1398] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:53:58.500617 24441 tablet_bootstrap.cc:492] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:53:58.502243 24441 tablet_bootstrap.cc:654] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
W20240502 01:53:58.502506 22053 leader_election.cc:341] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:58.502681 22053 leader_election.cc:302] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff
I20240502 01:53:58.503480 23967 raft_consensus.cc:2726] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:58.504841 24442 tablet_bootstrap.cc:492] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:58.505016 24442 ts_tablet_manager.cc:1367] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
I20240502 01:53:58.507090 24442 raft_consensus.cc:348] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:58.507325 24442 raft_consensus.cc:374] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:58.507427 24442 raft_consensus.cc:729] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:58.507694 24442 consensus_queue.cc:260] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:58.509183 24442 ts_tablet_manager.cc:1398] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:53:58.509704 24442 tablet_bootstrap.cc:492] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:53:58.511008 24442 tablet_bootstrap.cc:654] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:58.512552 24445 tablet_bootstrap.cc:492] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:53:58.512601 24446 tablet_bootstrap.cc:492] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:58.512770 24446 ts_tablet_manager.cc:1367] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.000s
I20240502 01:53:58.512805 24445 ts_tablet_manager.cc:1367] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.009s
I20240502 01:53:58.513689 24446 raft_consensus.cc:348] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:58.513679 24445 raft_consensus.cc:348] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:58.513901 24446 raft_consensus.cc:374] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:58.513921 24445 raft_consensus.cc:374] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:58.513995 24442 tablet_bootstrap.cc:492] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:53:58.513994 24446 raft_consensus.cc:729] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:58.513994 24445 raft_consensus.cc:729] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:53:58.514117 24442 ts_tablet_manager.cc:1367] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:53:58.514240 24446 consensus_queue.cc:260] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:58.514261 24445 consensus_queue.cc:260] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:58.514861 24446 ts_tablet_manager.cc:1398] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:58.514889 24445 ts_tablet_manager.cc:1398] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:58.515195 24446 tablet_bootstrap.cc:492] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:53:58.516492 24442 raft_consensus.cc:348] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:58.516733 24442 raft_consensus.cc:374] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:58.516811 24442 raft_consensus.cc:729] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:53:58.517652 24442 consensus_queue.cc:260] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:58.521693 24442 ts_tablet_manager.cc:1398] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:53:58.522851 24441 tablet_bootstrap.cc:492] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:53:58.523348 24441 ts_tablet_manager.cc:1367] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.005s
I20240502 01:53:58.523471 24446 tablet_bootstrap.cc:654] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:58.525527 24441 raft_consensus.cc:348] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:58.525802 24441 raft_consensus.cc:374] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:58.525910 24441 raft_consensus.cc:729] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:53:58.526319 24446 tablet_bootstrap.cc:492] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:53:58.526209 24441 consensus_queue.cc:260] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:58.526501 24446 ts_tablet_manager.cc:1367] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240502 01:53:58.527282 24446 raft_consensus.cc:348] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:58.527467 24446 raft_consensus.cc:374] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:58.527544 24446 raft_consensus.cc:729] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:53:58.527824 24441 ts_tablet_manager.cc:1398] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:58.527804 24446 consensus_queue.cc:260] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:58.528363 24446 ts_tablet_manager.cc:1398] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:58.531293 24436 raft_consensus.cc:482] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:58.531468 24436 raft_consensus.cc:504] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:58.532270 24436 leader_election.cc:288] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:58.532581 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2f4ea104d4409383e5eb32ca2f78b0" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:53:58.532752 21856 raft_consensus.cc:2454] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:58.533350 22184 leader_election.cc:302] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:58.533452 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2f4ea104d4409383e5eb32ca2f78b0" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:58.533692 24436 raft_consensus.cc:2781] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:58.533691 22119 raft_consensus.cc:2454] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:53:58.533814 24436 raft_consensus.cc:482] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:58.533897 24436 raft_consensus.cc:3037] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:58.536404 24436 raft_consensus.cc:504] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:58.537058 24436 leader_election.cc:288] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:58.537444 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2f4ea104d4409383e5eb32ca2f78b0" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:53:58.537621 21856 raft_consensus.cc:3037] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:58.537528 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2f4ea104d4409383e5eb32ca2f78b0" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:58.537758 22119 raft_consensus.cc:3037] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:58.539626 21856 raft_consensus.cc:2454] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:58.539674 22119 raft_consensus.cc:2454] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:53:58.540025 22184 leader_election.cc:302] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:58.540550 24436 raft_consensus.cc:2781] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:58.540722 24436 raft_consensus.cc:686] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:53:58.541074 24436 consensus_queue.cc:237] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:53:58.544999 21708 catalog_manager.cc:5484] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:53:58.737522 24409 raft_consensus.cc:482] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:58.737728 24409 raft_consensus.cc:504] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:58.738371 24409 leader_election.cc:288] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:58.738945 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b6dfe0e8da4352b136d6f97d473bad" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:58.739017 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b6dfe0e8da4352b136d6f97d473bad" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:53:58.739176 22251 raft_consensus.cc:2454] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:58.739248 22119 raft_consensus.cc:2454] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:53:58.739645 21924 leader_election.cc:302] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:58.739940 24409 raft_consensus.cc:2781] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:58.740072 24409 raft_consensus.cc:482] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:58.740159 24409 raft_consensus.cc:3037] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:58.742209 24409 raft_consensus.cc:504] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:58.742789 24409 leader_election.cc:288] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:53:58.743448 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b6dfe0e8da4352b136d6f97d473bad" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:58.743432 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b6dfe0e8da4352b136d6f97d473bad" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:53:58.743690 22119 raft_consensus.cc:3037] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:58.743685 22251 raft_consensus.cc:3037] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:58.746613 22119 raft_consensus.cc:2454] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:58.746626 22251 raft_consensus.cc:2454] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:53:58.747062 21924 leader_election.cc:302] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:58.747711 24409 raft_consensus.cc:2781] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:58.747885 24409 raft_consensus.cc:686] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:53:58.748243 24409 consensus_queue.cc:237] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:53:58.751557 21708 catalog_manager.cc:5484] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:53:58.802896 24384 raft_consensus.cc:482] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:58.803057 24384 raft_consensus.cc:504] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:58.803742 24384 leader_election.cc:288] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:58.804126 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d602fa045e4cfa9b698d3e92e5cf8f" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:53:58.804214 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d602fa045e4cfa9b698d3e92e5cf8f" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:53:58.804373 22251 raft_consensus.cc:2454] T 84d602fa045e4cfa9b698d3e92e5cf8f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:58.804472 21988 raft_consensus.cc:2454] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:53:58.804807 21792 leader_election.cc:302] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:58.805122 24384 raft_consensus.cc:2781] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:58.805225 24384 raft_consensus.cc:482] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:58.805317 24384 raft_consensus.cc:3037] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:58.807332 24384 raft_consensus.cc:504] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:58.807868 24384 leader_election.cc:288] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:53:58.808251 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d602fa045e4cfa9b698d3e92e5cf8f" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:53:58.808450 22251 raft_consensus.cc:3037] T 84d602fa045e4cfa9b698d3e92e5cf8f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:58.808435 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d602fa045e4cfa9b698d3e92e5cf8f" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:53:58.808678 21988 raft_consensus.cc:3037] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:58.810468 22251 raft_consensus.cc:2454] T 84d602fa045e4cfa9b698d3e92e5cf8f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:58.810679 21988 raft_consensus.cc:2454] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:53:58.810873 21792 leader_election.cc:302] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:53:58.811185 24384 raft_consensus.cc:2781] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:58.811321 24384 raft_consensus.cc:686] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:53:58.811614 24384 consensus_queue.cc:237] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:53:58.815392 21708 catalog_manager.cc:5484] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:53:59.056036 24456 consensus_queue.cc:1035] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:59.066989 24436 consensus_queue.cc:1035] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:59.164086 24444 consensus_queue.cc:1035] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:59.173676 24444 consensus_queue.cc:1035] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:59.223982 24461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:59.224359 24461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:59.232245 24461 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:53:59.272748 24384 consensus_queue.cc:1035] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:59.278198 24259 consensus_queue.cc:1035] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_19'
INSERT report
rows total: 2000
time total: 306.085 ms
time per row: 0.153043 ms
I20240502 01:53:59.725314 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.197:36677
--local_ip_for_outbound_sockets=127.14.212.197
--tserver_master_addrs=127.14.212.254:33157
--webserver_port=35037
--webserver_interface=127.14.212.197
--builtin_ntp_servers=127.14.212.212:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:54:00.095307 24521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:00.095620 24521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:00.095767 24521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:00.104425 24521 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:54:00.104606 24521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:00.104822 24521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.197
I20240502 01:54:00.116235 24521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.197:36677
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.212.197
--webserver_port=35037
--tserver_master_addrs=127.14.212.254:33157
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.197
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:00.117954 24521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:00.119944 24521 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:00.132421 24526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:00.132632 24527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:00.133476 24529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:00.134663 24521 server_base.cc:1034] running on GCE node
I20240502 01:54:00.135259 24521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:00.136211 24521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:00.137419 24521 hybrid_clock.cc:648] HybridClock initialized: now 1714614840137360 us; error 54 us; skew 500 ppm
I20240502 01:54:00.137787 24521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:00.140143 24521 webserver.cc:460] Webserver started at http://127.14.212.197:35037/ using document root <none> and password file <none>
I20240502 01:54:00.140872 24521 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:00.141034 24521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:00.146934 24521 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:54:00.150043 24535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:00.151417 24521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:54:00.151600 24521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
format_stamp: "Formatted at 2024-05-02 01:53:27 on dist-test-slave-mk1r"
I20240502 01:54:00.152586 24521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:00.189429 24521 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:00.190364 24521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:00.190666 24521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:00.191638 24521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:00.193248 24521 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:00.193326 24521 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:00.193451 24521 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:00.193508 24521 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:00.262841 24521 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.197:36677
I20240502 01:54:00.262939 24647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.197:36677 every 8 connection(s)
I20240502 01:54:00.264940 24521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:54:00.270527 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 24521
I20240502 01:54:00.272504 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.198:42375
--local_ip_for_outbound_sockets=127.14.212.198
--tserver_master_addrs=127.14.212.254:33157
--webserver_port=41119
--webserver_interface=127.14.212.198
--builtin_ntp_servers=127.14.212.212:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:54:00.286227 24648 heartbeater.cc:347] Connected to a master server at 127.14.212.254:33157
I20240502 01:54:00.286630 24648 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:00.287671 24648 heartbeater.cc:511] Master 127.14.212.254:33157 requested a full tablet report, sending...
I20240502 01:54:00.293833 21715 ts_manager.cc:194] Re-registered known tserver with Master: 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:00.294764 21715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.197:52899
W20240502 01:54:00.671227 24652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:00.671527 24652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:00.671628 24652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:00.680135 24652 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:54:00.680281 24652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:00.680413 24652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.198
I20240502 01:54:00.690876 24652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.198:42375
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.14.212.198
--webserver_port=41119
--tserver_master_addrs=127.14.212.254:33157
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.198
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:00.692406 24652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:00.694264 24652 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:00.706840 24659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:00.707016 24658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:00.707468 24661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:00.709687 24652 server_base.cc:1034] running on GCE node
I20240502 01:54:00.710279 24652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:00.711259 24652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:00.712466 24652 hybrid_clock.cc:648] HybridClock initialized: now 1714614840712409 us; error 54 us; skew 500 ppm
I20240502 01:54:00.712803 24652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:00.719879 24652 webserver.cc:460] Webserver started at http://127.14.212.198:41119/ using document root <none> and password file <none>
I20240502 01:54:00.720682 24652 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:00.720820 24652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:00.726835 24652 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 01:54:00.729809 24667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:00.731177 24652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:54:00.731343 24652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "28157bc3e678428fb5f93c077d528513"
format_stamp: "Formatted at 2024-05-02 01:53:28 on dist-test-slave-mk1r"
I20240502 01:54:00.732257 24652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:00.763003 24652 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:00.763890 24652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:00.764174 24652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:00.765324 24652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:00.767149 24652 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:00.767233 24652 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:00.767375 24652 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:00.767436 24652 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:00.828140 24652 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.198:42375
I20240502 01:54:00.828214 24779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.198:42375 every 8 connection(s)
I20240502 01:54:00.829941 24652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:54:00.833664 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 24652
I20240502 01:54:00.834442 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.199:35737
--local_ip_for_outbound_sockets=127.14.212.199
--tserver_master_addrs=127.14.212.254:33157
--webserver_port=41269
--webserver_interface=127.14.212.199
--builtin_ntp_servers=127.14.212.212:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:54:00.849978 24780 heartbeater.cc:347] Connected to a master server at 127.14.212.254:33157
I20240502 01:54:00.850353 24780 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:00.851361 24780 heartbeater.cc:511] Master 127.14.212.254:33157 requested a full tablet report, sending...
I20240502 01:54:00.853986 21715 ts_manager.cc:194] Re-registered known tserver with Master: 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:00.855019 21715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.198:49887
W20240502 01:54:01.204924 24784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:01.205232 24784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:01.205402 24784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:01.213403 24784 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:54:01.213569 24784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:01.213742 24784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.199
I20240502 01:54:01.224031 24784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.199:35737
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.14.212.199
--webserver_port=41269
--tserver_master_addrs=127.14.212.254:33157
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.199
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:01.225849 24784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:01.227802 24784 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:01.240783 24791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:01.241379 24790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:01.241717 24784 server_base.cc:1034] running on GCE node
W20240502 01:54:01.246217 24793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:01.247018 24784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:01.258857 24784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:01.265311 24784 hybrid_clock.cc:648] HybridClock initialized: now 1714614841265192 us; error 157 us; skew 500 ppm
I20240502 01:54:01.265799 24784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:01.272789 24784 webserver.cc:460] Webserver started at http://127.14.212.199:41269/ using document root <none> and password file <none>
I20240502 01:54:01.273620 24784 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:01.273736 24784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:01.279529 24784 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 01:54:01.282761 24800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:01.284198 24784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20240502 01:54:01.284384 24784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "d7449bb1df214bf4a5f93e4631312f08"
format_stamp: "Formatted at 2024-05-02 01:53:28 on dist-test-slave-mk1r"
I20240502 01:54:01.285336 24784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:01.297580 24648 heartbeater.cc:503] Master 127.14.212.254:33157 was elected leader, sending a full tablet report...
I20240502 01:54:01.315863 24784 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:01.316723 24784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:01.317059 24784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:01.318063 24784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:01.319594 24784 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:01.319675 24784 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:01.319809 24784 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:01.319865 24784 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:01.362084 24784 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.199:35737
I20240502 01:54:01.362133 24912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.199:35737 every 8 connection(s)
I20240502 01:54:01.363794 24784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:54:01.367986 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 24784
I20240502 01:54:01.368798 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.200:37097
--local_ip_for_outbound_sockets=127.14.212.200
--tserver_master_addrs=127.14.212.254:33157
--webserver_port=34041
--webserver_interface=127.14.212.200
--builtin_ntp_servers=127.14.212.212:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:54:01.379113 24913 heartbeater.cc:347] Connected to a master server at 127.14.212.254:33157
I20240502 01:54:01.379496 24913 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:01.380481 24913 heartbeater.cc:511] Master 127.14.212.254:33157 requested a full tablet report, sending...
I20240502 01:54:01.382119 21715 ts_manager.cc:194] Re-registered known tserver with Master: d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:01.383217 21715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.199:48599
W20240502 01:54:01.734808 24917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:01.735102 24917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:01.735241 24917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:01.744110 24917 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:54:01.744292 24917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:01.744459 24917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.200
I20240502 01:54:01.754508 24917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.200:37097
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.14.212.200
--webserver_port=34041
--tserver_master_addrs=127.14.212.254:33157
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.200
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:01.755985 24917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:01.757766 24917 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:01.772156 24923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:01.772385 24924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:01.773003 24926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:01.773330 24917 server_base.cc:1034] running on GCE node
I20240502 01:54:01.773928 24917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:01.775568 24917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:01.776835 24917 hybrid_clock.cc:648] HybridClock initialized: now 1714614841776784 us; error 54 us; skew 500 ppm
I20240502 01:54:01.777276 24917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:01.785231 24917 webserver.cc:460] Webserver started at http://127.14.212.200:34041/ using document root <none> and password file <none>
I20240502 01:54:01.786366 24917 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:01.786537 24917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:01.799085 24917 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.004s sys 0.004s
I20240502 01:54:01.802966 24932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:01.804951 24917 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20240502 01:54:01.805140 24917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "41c70f373c9d49a49891b03a12c6cb2b"
format_stamp: "Formatted at 2024-05-02 01:53:29 on dist-test-slave-mk1r"
I20240502 01:54:01.806352 24917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:01.829061 24917 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:01.829901 24917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:01.830197 24917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:01.831202 24917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:01.833056 24917 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:01.833145 24917 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:01.833298 24917 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:01.833359 24917 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:01.859488 24780 heartbeater.cc:503] Master 127.14.212.254:33157 was elected leader, sending a full tablet report...
I20240502 01:54:01.884562 24917 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.200:37097
I20240502 01:54:01.884616 25046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.200:37097 every 8 connection(s)
I20240502 01:54:01.886215 24917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:54:01.896672 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 24917
I20240502 01:54:01.900446 21715 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:01.902174 25047 heartbeater.cc:347] Connected to a master server at 127.14.212.254:33157
I20240502 01:54:01.902549 25047 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:01.903739 25047 heartbeater.cc:511] Master 127.14.212.254:33157 requested a full tablet report, sending...
I20240502 01:54:01.905591 21707 ts_manager.cc:194] Re-registered known tserver with Master: 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:01.906772 21707 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.200:35337
I20240502 01:54:01.933353 21835 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3c44e842924e62be7864dc3a8c7739 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=21148794be6b42c382fc735c63f91ac6]), partition=
I20240502 01:54:01.933815 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3c44e842924e62be7864dc3a8c7739. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:01.950016 24981 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3c44e842924e62be7864dc3a8c7739 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=21148794be6b42c382fc735c63f91ac6]), partition=
I20240502 01:54:01.951264 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3c44e842924e62be7864dc3a8c7739. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:01.954768 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3c44e842924e62be7864dc3a8c7739 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=21148794be6b42c382fc735c63f91ac6]), partition=
I20240502 01:54:01.956420 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3c44e842924e62be7864dc3a8c7739. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:01.961819 25064 tablet_bootstrap.cc:492] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:01.968062 25064 tablet_bootstrap.cc:654] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:01.973517 25064 tablet_bootstrap.cc:492] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:01.973830 25064 ts_tablet_manager.cc:1367] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 01:54:01.974740 25064 raft_consensus.cc:348] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:01.975111 25064 raft_consensus.cc:374] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:01.975281 25064 raft_consensus.cc:729] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:01.975652 25064 consensus_queue.cc:260] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:01.976691 25066 tablet_bootstrap.cc:492] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:01.977990 25064 ts_tablet_manager.cc:1398] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:54:01.982553 25066 tablet_bootstrap.cc:654] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:01.984169 25066 log.cc:824] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:01.995447 25066 tablet_bootstrap.cc:492] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:01.995954 25066 ts_tablet_manager.cc:1367] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.008s
I20240502 01:54:02.000399 25067 tablet_bootstrap.cc:492] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:02.007234 25066 raft_consensus.cc:348] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:02.007743 25066 raft_consensus.cc:374] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.007859 25066 raft_consensus.cc:729] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.008297 25067 tablet_bootstrap.cc:654] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.008579 25066 consensus_queue.cc:260] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:02.009660 25067 log.cc:824] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:02.009811 25047 heartbeater.cc:503] Master 127.14.212.254:33157 was elected leader, sending a full tablet report...
I20240502 01:54:02.010386 25066 ts_tablet_manager.cc:1398] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20240502 01:54:02.020287 25067 tablet_bootstrap.cc:492] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:02.020586 25067 ts_tablet_manager.cc:1367] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.000s
I20240502 01:54:02.027827 25067 raft_consensus.cc:348] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:02.028342 25067 raft_consensus.cc:374] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.028457 25067 raft_consensus.cc:729] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.029183 25067 consensus_queue.cc:260] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:02.030915 25067 ts_tablet_manager.cc:1398] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20240502 01:54:02.034962 24785 raft_consensus.cc:482] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:02.035181 24785 raft_consensus.cc:504] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:02.036209 24785 leader_election.cc:288] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:02.045286 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3c44e842924e62be7864dc3a8c7739" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:02.045843 24735 raft_consensus.cc:2454] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:54:02.046240 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3c44e842924e62be7864dc3a8c7739" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:02.046468 21791 leader_election.cc:302] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:02.046808 25001 raft_consensus.cc:2454] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:54:02.046835 24785 raft_consensus.cc:2781] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:02.046952 24785 raft_consensus.cc:482] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:02.047041 24785 raft_consensus.cc:3037] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.049306 24785 raft_consensus.cc:504] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:02.050503 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3c44e842924e62be7864dc3a8c7739" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:02.050756 24735 raft_consensus.cc:3037] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.051230 24785 leader_election.cc:288] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:02.053280 24735 raft_consensus.cc:2454] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:54:02.053233 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3c44e842924e62be7864dc3a8c7739" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:02.053471 25001 raft_consensus.cc:3037] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.053717 21791 leader_election.cc:302] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:02.054000 24785 raft_consensus.cc:2781] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:02.054167 24785 raft_consensus.cc:686] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:54:02.054700 24785 consensus_queue.cc:237] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:02.056578 25001 raft_consensus.cc:2454] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:54:02.058773 21715 catalog_manager.cc:5484] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
W20240502 01:54:02.088711 24781 tablet.cc:2369] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:54:02.138427 25048 tablet.cc:2369] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:54:02.166035 21715 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:02.178004 24735 raft_consensus.cc:1261] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:02.178602 25001 raft_consensus.cc:1261] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:02.179740 24259 consensus_queue.cc:1035] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:02.180506 24785 consensus_queue.cc:1035] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:02.192453 21715 catalog_manager.cc:5576] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375) does not have the latest schema for tablet 5b3c44e842924e62be7864dc3a8c7739 (table rebalancer_dynamic_table [id=21148794be6b42c382fc735c63f91ac6]). Expected version 1 got 0
I20240502 01:54:02.194597 25080 tablet.cc:1722] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:02.196615 25080 tablet.cc:1620] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77: MemRowSet was empty: no flush needed.
I20240502 01:54:02.197682 25081 tablet.cc:1722] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:02.198995 25081 tablet.cc:1620] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:02.200461 21707 catalog_manager.cc:5576] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375) does not have the latest schema for tablet 5b3c44e842924e62be7864dc3a8c7739 (table rebalancer_dynamic_table [id=21148794be6b42c382fc735c63f91ac6]). Expected version 1 got 0
I20240502 01:54:02.216434 21707 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=21148794be6b42c382fc735c63f91ac6] alter complete (version 1)
I20240502 01:54:02.222149 21715 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 5b3c44e842924e62be7864dc3a8c7739 (table rebalancer_dynamic_table [id=21148794be6b42c382fc735c63f91ac6]). Expected version 1 got 0
I20240502 01:54:02.227267 25086 tablet.cc:1722] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:02.228766 21715 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 5b3c44e842924e62be7864dc3a8c7739 (table rebalancer_dynamic_table [id=21148794be6b42c382fc735c63f91ac6]). Expected version 1 got 0
I20240502 01:54:02.229043 25086 tablet.cc:1620] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:02.326594 21715 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:02.335055 25080 tablet.cc:1722] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:02.335691 25080 tablet.cc:1620] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77: MemRowSet was empty: no flush needed.
I20240502 01:54:02.335744 25081 tablet.cc:1722] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:02.336231 25081 tablet.cc:1620] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:02.347782 25086 tablet.cc:1722] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:02.348341 25086 tablet.cc:1620] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:02.349613 21715 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=21148794be6b42c382fc735c63f91ac6] alter complete (version 2)
I20240502 01:54:02.386561 24913 heartbeater.cc:503] Master 127.14.212.254:33157 was elected leader, sending a full tablet report...
I20240502 01:54:02.399416 21707 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 01:54:02.399447 21715 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:02.399749 21707 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 01:54:02.405319 21707 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet b02ca6aec3774fe5a1aff6ee307cbad5
I20240502 01:54:02.406658 21707 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 95fa4e82b6a44b5bbc0f689a318090f7
I20240502 01:54:02.407193 21707 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet cddc036193de43c9b7569f664f3e4d7c
I20240502 01:54:02.407363 21835 tablet_service.cc:1514] Processing DeleteTablet for tablet b02ca6aec3774fe5a1aff6ee307cbad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:02.407634 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 95fa4e82b6a44b5bbc0f689a318090f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:02.408401 25097 tablet_replica.cc:330] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:02.408643 22098 tablet_service.cc:1514] Processing DeleteTablet for tablet cddc036193de43c9b7569f664f3e4d7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:02.408922 22230 tablet_service.cc:1514] Processing DeleteTablet for tablet b02ca6aec3774fe5a1aff6ee307cbad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:02.409509 21969 tablet_service.cc:1514] Processing DeleteTablet for tablet b02ca6aec3774fe5a1aff6ee307cbad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:02.409740 21968 tablet_service.cc:1514] Processing DeleteTablet for tablet 95fa4e82b6a44b5bbc0f689a318090f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:02.409579 25097 raft_consensus.cc:2229] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:02.408970 22229 tablet_service.cc:1514] Processing DeleteTablet for tablet cddc036193de43c9b7569f664f3e4d7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:02.410214 21837 tablet_service.cc:1514] Processing DeleteTablet for tablet 95fa4e82b6a44b5bbc0f689a318090f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:02.410398 25097 raft_consensus.cc:2258] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.410434 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet cddc036193de43c9b7569f664f3e4d7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:02.412003 25097 ts_tablet_manager.cc:1875] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.417860 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 03fb607ef0d9469daca1d6f3c5259fb2 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=ee5b662f59f047049d4a2a52984c0cfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:02.418514 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03fb607ef0d9469daca1d6f3c5259fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.419486 25097 ts_tablet_manager.cc:1888] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:02.419667 25097 log.cc:1196] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/b02ca6aec3774fe5a1aff6ee307cbad5
I20240502 01:54:02.420276 25097 ts_tablet_manager.cc:1909] T b02ca6aec3774fe5a1aff6ee307cbad5 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:02.422084 25097 tablet_replica.cc:330] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:02.422222 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet b02ca6aec3774fe5a1aff6ee307cbad5 (table rebalance_test_table_0 [id=14af009e70d54323b1c55d5cc89d84d5]) successfully deleted
I20240502 01:54:02.422345 25097 raft_consensus.cc:2229] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:02.422600 25097 raft_consensus.cc:2258] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.423429 25097 ts_tablet_manager.cc:1875] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.424283 25099 tablet_replica.cc:330] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:02.425130 25099 raft_consensus.cc:2229] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:02.425874 25099 raft_consensus.cc:2258] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.427881 25099 ts_tablet_manager.cc:1875] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.430253 25097 ts_tablet_manager.cc:1888] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:02.430418 25097 log.cc:1196] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/95fa4e82b6a44b5bbc0f689a318090f7
I20240502 01:54:02.430948 25097 ts_tablet_manager.cc:1909] T 95fa4e82b6a44b5bbc0f689a318090f7 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:02.432113 25097 tablet_replica.cc:330] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:02.432283 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 95fa4e82b6a44b5bbc0f689a318090f7 (table rebalance_test_table_0 [id=14af009e70d54323b1c55d5cc89d84d5]) successfully deleted
I20240502 01:54:02.432411 25097 raft_consensus.cc:2229] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:02.433022 25097 raft_consensus.cc:2258] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.433833 25097 ts_tablet_manager.cc:1875] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.436841 25099 ts_tablet_manager.cc:1888] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:02.437111 25099 log.cc:1196] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/b02ca6aec3774fe5a1aff6ee307cbad5
I20240502 01:54:02.437764 25099 ts_tablet_manager.cc:1909] T b02ca6aec3774fe5a1aff6ee307cbad5 P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:02.438304 25098 tablet_replica.cc:330] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:02.440563 25099 tablet_replica.cc:330] T cddc036193de43c9b7569f664f3e4d7c P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:02.440557 25097 ts_tablet_manager.cc:1888] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:02.440546 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet b02ca6aec3774fe5a1aff6ee307cbad5 (table rebalance_test_table_0 [id=14af009e70d54323b1c55d5cc89d84d5]) successfully deleted
I20240502 01:54:02.440686 25097 log.cc:1196] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/cddc036193de43c9b7569f664f3e4d7c
I20240502 01:54:02.440809 25099 raft_consensus.cc:2229] T cddc036193de43c9b7569f664f3e4d7c P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:02.441095 25099 raft_consensus.cc:2258] T cddc036193de43c9b7569f664f3e4d7c P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.441244 25097 ts_tablet_manager.cc:1909] T cddc036193de43c9b7569f664f3e4d7c P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:02.441893 25099 ts_tablet_manager.cc:1875] T cddc036193de43c9b7569f664f3e4d7c P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.442371 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet cddc036193de43c9b7569f664f3e4d7c (table rebalance_test_table_0 [id=14af009e70d54323b1c55d5cc89d84d5]) successfully deleted
I20240502 01:54:02.442703 25098 raft_consensus.cc:2229] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:02.443375 25098 raft_consensus.cc:2258] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.443715 25067 tablet_bootstrap.cc:492] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:02.445603 25067 tablet_bootstrap.cc:654] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.447526 25098 ts_tablet_manager.cc:1875] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.448014 25067 tablet_bootstrap.cc:492] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:02.448187 25067 ts_tablet_manager.cc:1367] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:54:02.449934 21968 tablet_service.cc:1467] Processing CreateTablet for tablet 03fb607ef0d9469daca1d6f3c5259fb2 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=ee5b662f59f047049d4a2a52984c0cfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:02.450441 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03fb607ef0d9469daca1d6f3c5259fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.454233 21707 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:02.457386 25067 raft_consensus.cc:348] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:02.457679 25067 raft_consensus.cc:374] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.457783 25067 raft_consensus.cc:729] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.458035 25067 consensus_queue.cc:260] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:02.461436 25067 ts_tablet_manager.cc:1398] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240502 01:54:02.463781 25080 tablet.cc:1722] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:02.465809 25086 tablet.cc:1722] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:02.466718 25081 tablet.cc:1722] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:02.471156 22230 tablet_service.cc:1467] Processing CreateTablet for tablet b0c930e4ad4544c18bad9c0d02a16184 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=ee5b662f59f047049d4a2a52984c0cfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:02.471702 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c930e4ad4544c18bad9c0d02a16184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.480456 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 03fb607ef0d9469daca1d6f3c5259fb2 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=ee5b662f59f047049d4a2a52984c0cfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:02.481338 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 0cce126a8ba64fb2b40c4ddd03982f19 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=ee5b662f59f047049d4a2a52984c0cfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:02.481866 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cce126a8ba64fb2b40c4ddd03982f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.482369 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03fb607ef0d9469daca1d6f3c5259fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.485734 24581 tablet_service.cc:1467] Processing CreateTablet for tablet 0cce126a8ba64fb2b40c4ddd03982f19 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=ee5b662f59f047049d4a2a52984c0cfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:02.486220 24581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cce126a8ba64fb2b40c4ddd03982f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.488507 24582 tablet_service.cc:1467] Processing CreateTablet for tablet b0c930e4ad4544c18bad9c0d02a16184 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=ee5b662f59f047049d4a2a52984c0cfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:02.489146 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c930e4ad4544c18bad9c0d02a16184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.490612 25100 tablet_replica.cc:330] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:02.491153 25100 raft_consensus.cc:2229] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:02.494596 25099 ts_tablet_manager.cc:1888] T cddc036193de43c9b7569f664f3e4d7c P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:02.491542 25100 raft_consensus.cc:2258] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.499691 21707 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=21148794be6b42c382fc735c63f91ac6] alter complete (version 3)
I20240502 01:54:02.501336 25099 log.cc:1196] T cddc036193de43c9b7569f664f3e4d7c P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/cddc036193de43c9b7569f664f3e4d7c
I20240502 01:54:02.501633 25100 ts_tablet_manager.cc:1875] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.502365 25099 ts_tablet_manager.cc:1909] T cddc036193de43c9b7569f664f3e4d7c P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:02.506677 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet cddc036193de43c9b7569f664f3e4d7c (table rebalance_test_table_0 [id=14af009e70d54323b1c55d5cc89d84d5]) successfully deleted
I20240502 01:54:02.505331 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 0cce126a8ba64fb2b40c4ddd03982f19 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=ee5b662f59f047049d4a2a52984c0cfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:02.512319 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cce126a8ba64fb2b40c4ddd03982f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.514129 25107 tablet_bootstrap.cc:492] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:02.515985 25100 ts_tablet_manager.cc:1888] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:02.514318 25098 ts_tablet_manager.cc:1888] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:02.516441 25098 log.cc:1196] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/95fa4e82b6a44b5bbc0f689a318090f7
I20240502 01:54:02.516486 25100 log.cc:1196] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/b02ca6aec3774fe5a1aff6ee307cbad5
I20240502 01:54:02.517125 25100 ts_tablet_manager.cc:1909] T b02ca6aec3774fe5a1aff6ee307cbad5 P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:02.517266 25098 ts_tablet_manager.cc:1909] T 95fa4e82b6a44b5bbc0f689a318090f7 P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:02.518935 25100 tablet_replica.cc:330] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:02.518991 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet b02ca6aec3774fe5a1aff6ee307cbad5 (table rebalance_test_table_0 [id=14af009e70d54323b1c55d5cc89d84d5]) successfully deleted
I20240502 01:54:02.519296 25100 raft_consensus.cc:2229] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:02.519604 25100 raft_consensus.cc:2258] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.518914 24848 tablet_service.cc:1467] Processing CreateTablet for tablet b0c930e4ad4544c18bad9c0d02a16184 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=ee5b662f59f047049d4a2a52984c0cfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:02.520509 25100 ts_tablet_manager.cc:1875] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.521095 25098 tablet_replica.cc:330] T cddc036193de43c9b7569f664f3e4d7c P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:02.521157 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 95fa4e82b6a44b5bbc0f689a318090f7 (table rebalance_test_table_0 [id=14af009e70d54323b1c55d5cc89d84d5]) successfully deleted
I20240502 01:54:02.521649 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c930e4ad4544c18bad9c0d02a16184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.524156 25098 raft_consensus.cc:2229] T cddc036193de43c9b7569f664f3e4d7c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:02.524413 25107 tablet_bootstrap.cc:654] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.524500 25098 raft_consensus.cc:2258] T cddc036193de43c9b7569f664f3e4d7c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.526194 25106 tablet_bootstrap.cc:492] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:02.528290 25107 tablet_bootstrap.cc:492] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:02.528442 25107 ts_tablet_manager.cc:1367] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240502 01:54:02.529249 25100 ts_tablet_manager.cc:1888] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:02.529418 25105 tablet_bootstrap.cc:492] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:02.529362 25107 raft_consensus.cc:348] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:02.529675 25107 raft_consensus.cc:374] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.529819 25107 raft_consensus.cc:729] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.530141 25107 consensus_queue.cc:260] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:02.530900 25107 ts_tablet_manager.cc:1398] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:54:02.531234 25105 tablet_bootstrap.cc:654] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.535115 25067 tablet_bootstrap.cc:492] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:02.536906 25067 tablet_bootstrap.cc:654] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.537123 25098 ts_tablet_manager.cc:1875] T cddc036193de43c9b7569f664f3e4d7c P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.544492 25111 tablet_bootstrap.cc:492] T b0c930e4ad4544c18bad9c0d02a16184 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:02.545032 25106 tablet_bootstrap.cc:654] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.548327 25111 tablet_bootstrap.cc:654] T b0c930e4ad4544c18bad9c0d02a16184 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.549099 25067 tablet_bootstrap.cc:492] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:02.549353 25067 ts_tablet_manager.cc:1367] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240502 01:54:02.550166 25067 raft_consensus.cc:348] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:02.550480 25067 raft_consensus.cc:374] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.550592 25067 raft_consensus.cc:729] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.550941 25067 consensus_queue.cc:260] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:02.552968 25100 log.cc:1196] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/95fa4e82b6a44b5bbc0f689a318090f7
I20240502 01:54:02.553417 25112 tablet_bootstrap.cc:492] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:02.553943 25067 ts_tablet_manager.cc:1398] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:02.555559 25100 ts_tablet_manager.cc:1909] T 95fa4e82b6a44b5bbc0f689a318090f7 P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:02.556805 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 95fa4e82b6a44b5bbc0f689a318090f7 (table rebalance_test_table_0 [id=14af009e70d54323b1c55d5cc89d84d5]) successfully deleted
I20240502 01:54:02.557632 25112 tablet_bootstrap.cc:654] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.549947 25111 log.cc:824] T b0c930e4ad4544c18bad9c0d02a16184 P d7449bb1df214bf4a5f93e4631312f08: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:02.558329 25105 tablet_bootstrap.cc:492] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:02.559870 25112 log.cc:824] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:02.562449 25105 ts_tablet_manager.cc:1367] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.000s
I20240502 01:54:02.563252 25112 tablet_bootstrap.cc:492] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:02.563584 25112 ts_tablet_manager.cc:1367] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240502 01:54:02.563730 25105 raft_consensus.cc:348] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:02.564024 25105 raft_consensus.cc:374] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.564121 25105 raft_consensus.cc:729] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.564471 25105 consensus_queue.cc:260] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:02.565466 25105 ts_tablet_manager.cc:1398] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:02.568323 25111 tablet_bootstrap.cc:492] T b0c930e4ad4544c18bad9c0d02a16184 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:02.568663 25111 ts_tablet_manager.cc:1367] T b0c930e4ad4544c18bad9c0d02a16184 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.000s
I20240502 01:54:02.572481 25098 ts_tablet_manager.cc:1888] T cddc036193de43c9b7569f664f3e4d7c P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:02.572676 25098 log.cc:1196] T cddc036193de43c9b7569f664f3e4d7c P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/cddc036193de43c9b7569f664f3e4d7c
I20240502 01:54:02.573189 25098 ts_tablet_manager.cc:1909] T cddc036193de43c9b7569f664f3e4d7c P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:02.574486 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet cddc036193de43c9b7569f664f3e4d7c (table rebalance_test_table_0 [id=14af009e70d54323b1c55d5cc89d84d5]) successfully deleted
I20240502 01:54:02.578846 25106 tablet_bootstrap.cc:492] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:02.579085 25106 ts_tablet_manager.cc:1367] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.008s
I20240502 01:54:02.580030 25106 raft_consensus.cc:348] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:02.580381 25106 raft_consensus.cc:374] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.580526 25106 raft_consensus.cc:729] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.580356 25112 raft_consensus.cc:348] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:02.580852 25112 raft_consensus.cc:374] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.580986 25112 raft_consensus.cc:729] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.581015 25106 consensus_queue.cc:260] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:02.581955 25112 consensus_queue.cc:260] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:02.583557 25111 raft_consensus.cc:348] T b0c930e4ad4544c18bad9c0d02a16184 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:02.584095 25111 raft_consensus.cc:374] T b0c930e4ad4544c18bad9c0d02a16184 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.584224 25111 raft_consensus.cc:729] T b0c930e4ad4544c18bad9c0d02a16184 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.585141 25111 consensus_queue.cc:260] T b0c930e4ad4544c18bad9c0d02a16184 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:02.590719 25112 ts_tablet_manager.cc:1398] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.027s user 0.015s sys 0.005s
I20240502 01:54:02.598592 25106 ts_tablet_manager.cc:1398] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240502 01:54:02.602312 25112 tablet_bootstrap.cc:492] T b0c930e4ad4544c18bad9c0d02a16184 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:02.604812 25112 tablet_bootstrap.cc:654] T b0c930e4ad4544c18bad9c0d02a16184 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.612017 25111 ts_tablet_manager.cc:1398] T b0c930e4ad4544c18bad9c0d02a16184 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.043s user 0.027s sys 0.011s
I20240502 01:54:02.613811 21715 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:02.614123 21715 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
W20240502 01:54:02.617336 24914 tablet.cc:2369] T b0c930e4ad4544c18bad9c0d02a16184 P d7449bb1df214bf4a5f93e4631312f08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:54:02.620820 25112 tablet_bootstrap.cc:492] T b0c930e4ad4544c18bad9c0d02a16184 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:02.620808 21715 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 5b3c44e842924e62be7864dc3a8c7739
I20240502 01:54:02.621024 25112 ts_tablet_manager.cc:1367] T b0c930e4ad4544c18bad9c0d02a16184 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.011s
I20240502 01:54:02.621955 25112 raft_consensus.cc:348] T b0c930e4ad4544c18bad9c0d02a16184 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:02.622156 25112 raft_consensus.cc:374] T b0c930e4ad4544c18bad9c0d02a16184 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.622264 25112 raft_consensus.cc:729] T b0c930e4ad4544c18bad9c0d02a16184 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.622275 24715 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3c44e842924e62be7864dc3a8c7739 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:41650
I20240502 01:54:02.622537 25112 consensus_queue.cc:260] T b0c930e4ad4544c18bad9c0d02a16184 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:02.623212 25112 ts_tablet_manager.cc:1398] T b0c930e4ad4544c18bad9c0d02a16184 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:02.623720 25112 tablet_bootstrap.cc:492] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:02.626580 25112 tablet_bootstrap.cc:654] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.628296 25122 tablet_replica.cc:330] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:54:02.628736 25122 raft_consensus.cc:2229] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:02.629086 25122 raft_consensus.cc:2258] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.630720 25122 ts_tablet_manager.cc:1875] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.634366 25112 tablet_bootstrap.cc:492] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:02.634627 25112 ts_tablet_manager.cc:1367] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20240502 01:54:02.635496 25112 raft_consensus.cc:348] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:02.635756 25112 raft_consensus.cc:374] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.635854 25112 raft_consensus.cc:729] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.636162 25112 consensus_queue.cc:260] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:02.637627 25112 ts_tablet_manager.cc:1398] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 01:54:02.640497 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3c44e842924e62be7864dc3a8c7739 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:02.640811 25097 tablet_replica.cc:330] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:02.641189 25097 raft_consensus.cc:2229] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:02.641624 25097 raft_consensus.cc:2258] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.642855 25097 ts_tablet_manager.cc:1875] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.645102 25122 ts_tablet_manager.cc:1888] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:02.645373 25122 log.cc:1196] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/5b3c44e842924e62be7864dc3a8c7739
I20240502 01:54:02.646046 25122 ts_tablet_manager.cc:1909] T 5b3c44e842924e62be7864dc3a8c7739 P 28157bc3e678428fb5f93c077d528513: Deleting consensus metadata
I20240502 01:54:02.645910 24981 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3c44e842924e62be7864dc3a8c7739 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:39964
I20240502 01:54:02.648061 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet 5b3c44e842924e62be7864dc3a8c7739 (table rebalancer_dynamic_table_new_name [id=21148794be6b42c382fc735c63f91ac6]) successfully deleted
I20240502 01:54:02.649271 25097 ts_tablet_manager.cc:1888] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:02.649422 25097 log.cc:1196] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/5b3c44e842924e62be7864dc3a8c7739
I20240502 01:54:02.649842 25097 ts_tablet_manager.cc:1909] T 5b3c44e842924e62be7864dc3a8c7739 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:02.650815 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 5b3c44e842924e62be7864dc3a8c7739 (table rebalancer_dynamic_table_new_name [id=21148794be6b42c382fc735c63f91ac6]) successfully deleted
I20240502 01:54:02.651038 25125 tablet_replica.cc:330] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:54:02.651459 25125 raft_consensus.cc:2229] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:02.651759 25125 raft_consensus.cc:2258] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.653133 25125 ts_tablet_manager.cc:1875] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.659060 25125 ts_tablet_manager.cc:1888] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:02.659211 25125 log.cc:1196] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/5b3c44e842924e62be7864dc3a8c7739
I20240502 01:54:02.659690 25125 ts_tablet_manager.cc:1909] T 5b3c44e842924e62be7864dc3a8c7739 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting consensus metadata
I20240502 01:54:02.662246 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet 5b3c44e842924e62be7864dc3a8c7739 (table rebalancer_dynamic_table_new_name [id=21148794be6b42c382fc735c63f91ac6]) successfully deleted
I20240502 01:54:02.708868 24346 raft_consensus.cc:482] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:02.709146 24346 raft_consensus.cc:504] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:02.710124 24346 leader_election.cc:288] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:02.711380 25075 raft_consensus.cc:482] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:02.711585 25075 raft_consensus.cc:504] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:02.712730 25075 leader_election.cc:288] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:02.717538 25071 raft_consensus.cc:482] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:02.717783 25071 raft_consensus.cc:504] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:02.720316 25071 leader_election.cc:288] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:02.750948 21706 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:02.759217 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c930e4ad4544c18bad9c0d02a16184" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:02.760001 24868 raft_consensus.cc:2454] T b0c930e4ad4544c18bad9c0d02a16184 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:54:02.760916 22185 leader_election.cc:302] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:02.761375 24456 raft_consensus.cc:2781] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:02.761509 24456 raft_consensus.cc:482] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:02.761600 24456 raft_consensus.cc:3037] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.763087 24715 tablet_service.cc:1467] Processing CreateTablet for tablet f48d8c7e94614d30950e418f2017ac21 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a71d72fbe2fb496e83394e55a713e713]), partition=
I20240502 01:54:02.763252 24848 tablet_service.cc:1467] Processing CreateTablet for tablet f48d8c7e94614d30950e418f2017ac21 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a71d72fbe2fb496e83394e55a713e713]), partition=
I20240502 01:54:02.763625 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f48d8c7e94614d30950e418f2017ac21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.763638 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f48d8c7e94614d30950e418f2017ac21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.764559 24456 raft_consensus.cc:504] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:02.765427 24456 leader_election.cc:288] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:02.766026 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c930e4ad4544c18bad9c0d02a16184" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:02.766242 24868 raft_consensus.cc:3037] T b0c930e4ad4544c18bad9c0d02a16184 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.767022 24981 tablet_service.cc:1467] Processing CreateTablet for tablet f48d8c7e94614d30950e418f2017ac21 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a71d72fbe2fb496e83394e55a713e713]), partition=
I20240502 01:54:02.767474 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f48d8c7e94614d30950e418f2017ac21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.771168 24868 raft_consensus.cc:2454] T b0c930e4ad4544c18bad9c0d02a16184 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:54:02.772356 22185 leader_election.cc:302] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, d7449bb1df214bf4a5f93e4631312f08; no voters:
W20240502 01:54:02.772513 24649 tablet.cc:2369] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:54:02.772716 24456 raft_consensus.cc:2781] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:02.772848 24456 raft_consensus.cc:686] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:54:02.773969 24456 consensus_queue.cc:237] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:02.780618 25067 tablet_bootstrap.cc:492] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:02.782274 25067 tablet_bootstrap.cc:654] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.786667 25116 raft_consensus.cc:482] T b0c930e4ad4544c18bad9c0d02a16184 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:02.786999 25116 raft_consensus.cc:504] T b0c930e4ad4544c18bad9c0d02a16184 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:02.790023 25116 leader_election.cc:288] T b0c930e4ad4544c18bad9c0d02a16184 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:02.794241 25111 tablet_bootstrap.cc:492] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:02.796411 25111 tablet_bootstrap.cc:654] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.799983 25071 raft_consensus.cc:482] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:02.800227 25071 raft_consensus.cc:504] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:02.801249 25071 leader_election.cc:288] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:02.820560 21706 catalog_manager.cc:5484] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:02.823503 25152 tablet_bootstrap.cc:492] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:02.829838 25152 tablet_bootstrap.cc:654] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.834729 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c930e4ad4544c18bad9c0d02a16184" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:02.835109 24868 raft_consensus.cc:2379] T b0c930e4ad4544c18bad9c0d02a16184 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c64d40b5d7644bd8b29feaa80eab72f in current term 1: Already voted for candidate c2a28baa7ee2469294d94b7fe5455cff in this term.
I20240502 01:54:02.853273 25067 tablet_bootstrap.cc:492] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:02.853506 25067 ts_tablet_manager.cc:1367] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.073s user 0.004s sys 0.008s
I20240502 01:54:02.854566 25067 raft_consensus.cc:348] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:02.854836 25067 raft_consensus.cc:374] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.854934 25067 raft_consensus.cc:729] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.855276 25067 consensus_queue.cc:260] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:02.856640 25152 tablet_bootstrap.cc:492] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:02.856907 25152 ts_tablet_manager.cc:1367] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.003s
I20240502 01:54:02.858459 25067 ts_tablet_manager.cc:1398] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:02.864706 25111 tablet_bootstrap.cc:492] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:02.868968 25111 ts_tablet_manager.cc:1367] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.075s user 0.008s sys 0.005s
I20240502 01:54:02.868961 25152 raft_consensus.cc:348] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:02.869354 25152 raft_consensus.cc:374] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.869501 25152 raft_consensus.cc:729] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.871340 25111 raft_consensus.cc:348] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:02.869771 25152 consensus_queue.cc:260] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:02.872510 25111 raft_consensus.cc:374] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.872707 25111 raft_consensus.cc:729] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.873101 25111 consensus_queue.cc:260] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:02.874369 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c930e4ad4544c18bad9c0d02a16184" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:02.874881 24603 raft_consensus.cc:2454] T b0c930e4ad4544c18bad9c0d02a16184 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:54:02.876031 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c930e4ad4544c18bad9c0d02a16184" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:02.876224 24603 raft_consensus.cc:3037] T b0c930e4ad4544c18bad9c0d02a16184 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.877460 25152 ts_tablet_manager.cc:1398] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240502 01:54:02.881529 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fb607ef0d9469daca1d6f3c5259fb2" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:54:02.881778 21988 raft_consensus.cc:2454] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:02.882315 24603 raft_consensus.cc:2454] T b0c930e4ad4544c18bad9c0d02a16184 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:54:02.885867 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c930e4ad4544c18bad9c0d02a16184" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:02.887698 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cce126a8ba64fb2b40c4ddd03982f19" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:02.888031 24603 raft_consensus.cc:2454] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:02.888505 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fb607ef0d9469daca1d6f3c5259fb2" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:02.888779 24603 raft_consensus.cc:2454] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:02.889384 24538 leader_election.cc:302] T b0c930e4ad4544c18bad9c0d02a16184 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: c2a28baa7ee2469294d94b7fe5455cff, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:02.890640 25116 raft_consensus.cc:2726] T b0c930e4ad4544c18bad9c0d02a16184 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:02.891614 25111 ts_tablet_manager.cc:1398] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240502 01:54:02.899242 24668 leader_election.cc:302] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:54:02.898769 24669 leader_election.cc:302] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:54:02.899811 25071 raft_consensus.cc:2781] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:02.899943 25071 raft_consensus.cc:482] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:02.900076 25071 raft_consensus.cc:3037] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.903249 25071 raft_consensus.cc:504] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:02.903954 25071 leader_election.cc:288] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:02.904783 25071 raft_consensus.cc:2781] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:02.904933 25071 raft_consensus.cc:482] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:02.905043 25071 raft_consensus.cc:3037] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.905441 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fb607ef0d9469daca1d6f3c5259fb2" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:02.905674 24603 raft_consensus.cc:3037] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.907770 24603 raft_consensus.cc:2454] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:02.908560 25071 raft_consensus.cc:504] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:02.908946 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fb607ef0d9469daca1d6f3c5259fb2" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:54:02.909162 21988 raft_consensus.cc:3037] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.909314 25071 leader_election.cc:288] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:02.908632 24669 leader_election.cc:302] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:54:02.910095 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cce126a8ba64fb2b40c4ddd03982f19" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:02.910287 24603 raft_consensus.cc:3037] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.912070 21988 raft_consensus.cc:2454] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:02.913915 21715 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 01:54:02.914166 21715 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 01:54:02.917621 24603 raft_consensus.cc:2454] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:02.918150 25071 raft_consensus.cc:2781] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:02.918643 24669 leader_election.cc:302] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:54:02.925530 21715 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet ae6c263bd6ab4cc48e589c8a18ef19a4
I20240502 01:54:02.926635 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet ae6c263bd6ab4cc48e589c8a18ef19a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:02.927179 25097 tablet_replica.cc:330] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:02.927455 25097 raft_consensus.cc:2229] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:02.927872 25097 raft_consensus.cc:2258] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.927456 21715 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 395c97e04948487cb64b0f3a4bf6f53c
I20240502 01:54:02.928781 25097 ts_tablet_manager.cc:1875] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.928810 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 395c97e04948487cb64b0f3a4bf6f53c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:02.928808 21968 tablet_service.cc:1514] Processing DeleteTablet for tablet ae6c263bd6ab4cc48e589c8a18ef19a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:02.929011 21715 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet a65fc966b76c4abe9fda77fe8938e4b9
I20240502 01:54:02.929129 25100 tablet_replica.cc:330] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:02.929350 25100 raft_consensus.cc:2229] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:02.929658 25100 raft_consensus.cc:2258] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.929752 21968 tablet_service.cc:1514] Processing DeleteTablet for tablet a65fc966b76c4abe9fda77fe8938e4b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:02.930473 25100 ts_tablet_manager.cc:1875] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.933454 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 395c97e04948487cb64b0f3a4bf6f53c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:02.933501 22230 tablet_service.cc:1514] Processing DeleteTablet for tablet ae6c263bd6ab4cc48e589c8a18ef19a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:02.933764 22229 tablet_service.cc:1514] Processing DeleteTablet for tablet 395c97e04948487cb64b0f3a4bf6f53c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:02.933969 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet a65fc966b76c4abe9fda77fe8938e4b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:02.934230 25098 tablet_replica.cc:330] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:02.934440 25098 raft_consensus.cc:2229] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:02.934814 25098 raft_consensus.cc:2258] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.935597 25098 ts_tablet_manager.cc:1875] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.937043 25099 tablet_replica.cc:330] T ae6c263bd6ab4cc48e589c8a18ef19a4 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:02.937281 25099 raft_consensus.cc:2229] T ae6c263bd6ab4cc48e589c8a18ef19a4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:02.937553 25099 raft_consensus.cc:2258] T ae6c263bd6ab4cc48e589c8a18ef19a4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.938297 25099 ts_tablet_manager.cc:1875] T ae6c263bd6ab4cc48e589c8a18ef19a4 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.939908 25097 ts_tablet_manager.cc:1888] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:02.941298 25097 log.cc:1196] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/ae6c263bd6ab4cc48e589c8a18ef19a4
I20240502 01:54:02.941807 25097 ts_tablet_manager.cc:1909] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:02.943950 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet a65fc966b76c4abe9fda77fe8938e4b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:02.944396 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cce126a8ba64fb2b40c4ddd03982f19" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:02.944777 24603 raft_consensus.cc:2379] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c64840657cdf4b93980bd34120c94d9a in current term 1: Already voted for candidate 28157bc3e678428fb5f93c077d528513 in this term.
I20240502 01:54:02.946664 25097 tablet_replica.cc:330] T 395c97e04948487cb64b0f3a4bf6f53c P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:02.946916 25097 raft_consensus.cc:2229] T 395c97e04948487cb64b0f3a4bf6f53c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:02.947186 25097 raft_consensus.cc:2258] T 395c97e04948487cb64b0f3a4bf6f53c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.947953 25097 ts_tablet_manager.cc:1875] T 395c97e04948487cb64b0f3a4bf6f53c P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.950376 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet ae6c263bd6ab4cc48e589c8a18ef19a4 (table rebalance_test_table_2 [id=c9e840e3d5c343ff9dac7e30dd1febee]) successfully deleted
I20240502 01:54:02.970793 25098 ts_tablet_manager.cc:1888] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:02.971905 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cce126a8ba64fb2b40c4ddd03982f19" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:54:02.972311 22119 raft_consensus.cc:2454] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:02.972613 22117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cce126a8ba64fb2b40c4ddd03982f19" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:54:02.972826 22117 raft_consensus.cc:3037] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.973400 25098 log.cc:1196] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/395c97e04948487cb64b0f3a4bf6f53c
I20240502 01:54:02.973855 25100 ts_tablet_manager.cc:1888] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:02.974066 25098 ts_tablet_manager.cc:1909] T 395c97e04948487cb64b0f3a4bf6f53c P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:02.974104 25100 log.cc:1196] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/ae6c263bd6ab4cc48e589c8a18ef19a4
I20240502 01:54:02.974754 25100 ts_tablet_manager.cc:1909] T ae6c263bd6ab4cc48e589c8a18ef19a4 P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:02.975360 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 395c97e04948487cb64b0f3a4bf6f53c (table rebalance_test_table_2 [id=c9e840e3d5c343ff9dac7e30dd1febee]) successfully deleted
I20240502 01:54:02.976672 25100 tablet_replica.cc:330] T a65fc966b76c4abe9fda77fe8938e4b9 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:02.977052 25100 raft_consensus.cc:2229] T a65fc966b76c4abe9fda77fe8938e4b9 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:02.977422 25100 raft_consensus.cc:2258] T a65fc966b76c4abe9fda77fe8938e4b9 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.977869 25099 ts_tablet_manager.cc:1888] T ae6c263bd6ab4cc48e589c8a18ef19a4 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:02.977849 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet ae6c263bd6ab4cc48e589c8a18ef19a4 (table rebalance_test_table_2 [id=c9e840e3d5c343ff9dac7e30dd1febee]) successfully deleted
I20240502 01:54:02.978197 25099 log.cc:1196] T ae6c263bd6ab4cc48e589c8a18ef19a4 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/ae6c263bd6ab4cc48e589c8a18ef19a4
W20240502 01:54:02.979108 24669 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.212.197:36677, user_credentials={real_user=slave}} blocked reactor thread for 60482.8us
I20240502 01:54:02.979950 25097 ts_tablet_manager.cc:1888] T 395c97e04948487cb64b0f3a4bf6f53c P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:02.980125 25097 log.cc:1196] T 395c97e04948487cb64b0f3a4bf6f53c P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/395c97e04948487cb64b0f3a4bf6f53c
I20240502 01:54:02.980576 25097 ts_tablet_manager.cc:1909] T 395c97e04948487cb64b0f3a4bf6f53c P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:02.981771 22117 raft_consensus.cc:2454] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:02.982879 25099 ts_tablet_manager.cc:1909] T ae6c263bd6ab4cc48e589c8a18ef19a4 P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:02.984380 25099 tablet_replica.cc:330] T 395c97e04948487cb64b0f3a4bf6f53c P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:02.984470 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet ae6c263bd6ab4cc48e589c8a18ef19a4 (table rebalance_test_table_2 [id=c9e840e3d5c343ff9dac7e30dd1febee]) successfully deleted
I20240502 01:54:02.985121 25099 raft_consensus.cc:2229] T 395c97e04948487cb64b0f3a4bf6f53c P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:02.985901 25099 raft_consensus.cc:2258] T 395c97e04948487cb64b0f3a4bf6f53c P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.986526 25097 tablet_replica.cc:330] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:02.987215 25099 ts_tablet_manager.cc:1875] T 395c97e04948487cb64b0f3a4bf6f53c P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.987226 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 395c97e04948487cb64b0f3a4bf6f53c (table rebalance_test_table_2 [id=c9e840e3d5c343ff9dac7e30dd1febee]) successfully deleted
I20240502 01:54:02.988359 25174 raft_consensus.cc:2781] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:02.989720 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cce126a8ba64fb2b40c4ddd03982f19" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:02.989720 25174 raft_consensus.cc:686] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:02.990471 25097 raft_consensus.cc:2229] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:02.990993 25100 ts_tablet_manager.cc:1875] T a65fc966b76c4abe9fda77fe8938e4b9 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.990820 25174 consensus_queue.cc:237] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:02.990952 25097 raft_consensus.cc:2258] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.992094 25097 ts_tablet_manager.cc:1875] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.994848 22054 leader_election.cc:302] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513
I20240502 01:54:02.995226 25075 raft_consensus.cc:2726] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:02.995422 25071 raft_consensus.cc:686] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:02.995687 25071 consensus_queue.cc:237] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:03.002465 25097 ts_tablet_manager.cc:1888] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:03.002671 25097 log.cc:1196] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/a65fc966b76c4abe9fda77fe8938e4b9
I20240502 01:54:03.003180 25097 ts_tablet_manager.cc:1909] T a65fc966b76c4abe9fda77fe8938e4b9 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:03.004240 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet a65fc966b76c4abe9fda77fe8938e4b9 (table rebalance_test_table_2 [id=c9e840e3d5c343ff9dac7e30dd1febee]) successfully deleted
I20240502 01:54:03.005676 25099 ts_tablet_manager.cc:1888] T 395c97e04948487cb64b0f3a4bf6f53c P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:03.005879 25099 log.cc:1196] T 395c97e04948487cb64b0f3a4bf6f53c P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/395c97e04948487cb64b0f3a4bf6f53c
I20240502 01:54:03.006337 25099 ts_tablet_manager.cc:1909] T 395c97e04948487cb64b0f3a4bf6f53c P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:03.007241 25100 ts_tablet_manager.cc:1888] T a65fc966b76c4abe9fda77fe8938e4b9 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:03.007403 25100 log.cc:1196] T a65fc966b76c4abe9fda77fe8938e4b9 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/a65fc966b76c4abe9fda77fe8938e4b9
I20240502 01:54:03.007562 25099 tablet_replica.cc:330] T a65fc966b76c4abe9fda77fe8938e4b9 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:03.007334 21712 catalog_manager.cc:5484] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:03.007833 25100 ts_tablet_manager.cc:1909] T a65fc966b76c4abe9fda77fe8938e4b9 P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:03.007876 25099 raft_consensus.cc:2229] T a65fc966b76c4abe9fda77fe8938e4b9 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.008164 25099 raft_consensus.cc:2258] T a65fc966b76c4abe9fda77fe8938e4b9 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.008702 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 395c97e04948487cb64b0f3a4bf6f53c (table rebalance_test_table_2 [id=c9e840e3d5c343ff9dac7e30dd1febee]) successfully deleted
I20240502 01:54:03.008973 25099 ts_tablet_manager.cc:1875] T a65fc966b76c4abe9fda77fe8938e4b9 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:03.009071 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet a65fc966b76c4abe9fda77fe8938e4b9 (table rebalance_test_table_2 [id=c9e840e3d5c343ff9dac7e30dd1febee]) successfully deleted
I20240502 01:54:03.014159 21712 catalog_manager.cc:5484] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:03.015627 25099 ts_tablet_manager.cc:1888] T a65fc966b76c4abe9fda77fe8938e4b9 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:03.015780 25099 log.cc:1196] T a65fc966b76c4abe9fda77fe8938e4b9 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/a65fc966b76c4abe9fda77fe8938e4b9
I20240502 01:54:03.016232 25099 ts_tablet_manager.cc:1909] T a65fc966b76c4abe9fda77fe8938e4b9 P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:03.017272 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet a65fc966b76c4abe9fda77fe8938e4b9 (table rebalance_test_table_2 [id=c9e840e3d5c343ff9dac7e30dd1febee]) successfully deleted
I20240502 01:54:03.072803 24581 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:03.075491 21968 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:03.089862 22231 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:03.094305 24981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:03.105521 24715 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:03.110183 24848 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:03.135640 21836 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:03.157382 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:03.190860 25071 raft_consensus.cc:482] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:03.191063 25071 raft_consensus.cc:504] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:03.192003 25071 leader_election.cc:288] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:03.205430 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f48d8c7e94614d30950e418f2017ac21" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:03.205739 25001 raft_consensus.cc:2454] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:03.206276 24671 leader_election.cc:302] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:03.206748 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f48d8c7e94614d30950e418f2017ac21" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:03.207006 24868 raft_consensus.cc:2454] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:03.207713 25071 raft_consensus.cc:2781] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:03.207832 25071 raft_consensus.cc:482] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:03.207948 25071 raft_consensus.cc:3037] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:03.210808 25071 raft_consensus.cc:504] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:03.211997 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f48d8c7e94614d30950e418f2017ac21" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:03.212060 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f48d8c7e94614d30950e418f2017ac21" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:03.212266 25001 raft_consensus.cc:3037] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:03.212275 24868 raft_consensus.cc:3037] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:03.215306 24868 raft_consensus.cc:2454] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:03.215706 25001 raft_consensus.cc:2454] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:03.215829 24669 leader_election.cc:302] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:03.216159 25174 raft_consensus.cc:2781] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:03.216320 25174 raft_consensus.cc:686] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:03.216586 25174 consensus_queue.cc:237] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:03.218899 25071 leader_election.cc:288] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:03.219945 21712 catalog_manager.cc:5484] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:03.220465 24346 consensus_queue.cc:1035] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:03.231132 24346 consensus_queue.cc:1035] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:03.388481 21712 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:03.398634 24868 raft_consensus.cc:1261] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:03.399029 25001 raft_consensus.cc:1261] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:03.399567 25071 consensus_queue.cc:1035] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:03.399912 25174 consensus_queue.cc:1035] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:03.406615 25071 consensus_queue.cc:1035] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:03.417917 21712 catalog_manager.cc:5576] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737) does not have the latest schema for tablet f48d8c7e94614d30950e418f2017ac21 (table rebalancer_dynamic_table [id=a71d72fbe2fb496e83394e55a713e713]). Expected version 1 got 0
I20240502 01:54:03.423691 25196 mvcc.cc:205] Tried to move back new op lower bound from 7023062398552444928 to 7023062397817704448. Current Snapshot: MvccSnapshot[applied={T|T < 7023062398552444928}]
I20240502 01:54:03.424110 25228 tablet.cc:1722] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:03.427556 25228 tablet.cc:1620] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:03.428654 21715 catalog_manager.cc:5576] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737) does not have the latest schema for tablet f48d8c7e94614d30950e418f2017ac21 (table rebalancer_dynamic_table [id=a71d72fbe2fb496e83394e55a713e713]). Expected version 1 got 0
I20240502 01:54:03.433151 21715 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 01:54:03.433477 21715 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 01:54:03.437589 25227 tablet.cc:1722] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:03.438407 25227 tablet.cc:1620] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:03.440901 21715 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 8e18fd328b0a45b98eadddb574737672
I20240502 01:54:03.442317 21968 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e18fd328b0a45b98eadddb574737672 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:03.442708 25100 tablet_replica.cc:330] T 8e18fd328b0a45b98eadddb574737672 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:03.442960 25100 raft_consensus.cc:2229] T 8e18fd328b0a45b98eadddb574737672 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.443352 25100 raft_consensus.cc:2258] T 8e18fd328b0a45b98eadddb574737672 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.443552 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e18fd328b0a45b98eadddb574737672 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:03.444190 25100 ts_tablet_manager.cc:1875] T 8e18fd328b0a45b98eadddb574737672 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:03.444247 25097 tablet_replica.cc:330] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:03.444525 25097 raft_consensus.cc:2229] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:03.444795 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e18fd328b0a45b98eadddb574737672 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:03.445073 25098 tablet_replica.cc:330] T 8e18fd328b0a45b98eadddb574737672 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:03.445070 21715 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 5ccc8d73d1ea4539a170022fe6ad890e
I20240502 01:54:03.445307 25098 raft_consensus.cc:2229] T 8e18fd328b0a45b98eadddb574737672 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.445576 25098 raft_consensus.cc:2258] T 8e18fd328b0a45b98eadddb574737672 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.446085 21968 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ccc8d73d1ea4539a170022fe6ad890e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:03.446400 25098 ts_tablet_manager.cc:1875] T 8e18fd328b0a45b98eadddb574737672 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:03.446874 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ccc8d73d1ea4539a170022fe6ad890e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:03.447083 21715 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 8eb056b49f4d44869749288abfd50d84
I20240502 01:54:03.447995 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb056b49f4d44869749288abfd50d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:03.448808 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb056b49f4d44869749288abfd50d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:03.449385 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ccc8d73d1ea4539a170022fe6ad890e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:03.449426 25097 raft_consensus.cc:2258] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.449805 22229 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb056b49f4d44869749288abfd50d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:03.450255 25097 ts_tablet_manager.cc:1875] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:03.452232 21715 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet f48d8c7e94614d30950e418f2017ac21 (table rebalancer_dynamic_table [id=a71d72fbe2fb496e83394e55a713e713]). Expected version 1 got 0
I20240502 01:54:03.452473 25100 ts_tablet_manager.cc:1888] T 8e18fd328b0a45b98eadddb574737672 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:03.452773 25100 log.cc:1196] T 8e18fd328b0a45b98eadddb574737672 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/8e18fd328b0a45b98eadddb574737672
I20240502 01:54:03.453048 25099 tablet_replica.cc:330] T 5ccc8d73d1ea4539a170022fe6ad890e P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:03.453316 25099 raft_consensus.cc:2229] T 5ccc8d73d1ea4539a170022fe6ad890e P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.453488 25100 ts_tablet_manager.cc:1909] T 8e18fd328b0a45b98eadddb574737672 P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:03.453606 25099 raft_consensus.cc:2258] T 5ccc8d73d1ea4539a170022fe6ad890e P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.454422 25099 ts_tablet_manager.cc:1875] T 5ccc8d73d1ea4539a170022fe6ad890e P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:03.454888 25100 tablet_replica.cc:330] T 5ccc8d73d1ea4539a170022fe6ad890e P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:03.455003 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 8e18fd328b0a45b98eadddb574737672 (table rebalance_test_table_4 [id=0bafdd6983184b96aa892f802d62ebbc]) successfully deleted
I20240502 01:54:03.455010 25071 consensus_queue.cc:1035] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:03.455302 25100 raft_consensus.cc:2229] T 5ccc8d73d1ea4539a170022fe6ad890e P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.455570 25100 raft_consensus.cc:2258] T 5ccc8d73d1ea4539a170022fe6ad890e P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.458127 25100 ts_tablet_manager.cc:1875] T 5ccc8d73d1ea4539a170022fe6ad890e P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:03.464187 21715 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=a71d72fbe2fb496e83394e55a713e713] alter complete (version 1)
I20240502 01:54:03.465219 25098 ts_tablet_manager.cc:1888] T 8e18fd328b0a45b98eadddb574737672 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:03.465548 25098 log.cc:1196] T 8e18fd328b0a45b98eadddb574737672 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/8e18fd328b0a45b98eadddb574737672
I20240502 01:54:03.466104 25098 ts_tablet_manager.cc:1909] T 8e18fd328b0a45b98eadddb574737672 P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:03.468070 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 8e18fd328b0a45b98eadddb574737672 (table rebalance_test_table_4 [id=0bafdd6983184b96aa892f802d62ebbc]) successfully deleted
I20240502 01:54:03.468269 25098 tablet_replica.cc:330] T 8eb056b49f4d44869749288abfd50d84 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:03.468531 25098 raft_consensus.cc:2229] T 8eb056b49f4d44869749288abfd50d84 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.468905 25098 raft_consensus.cc:2258] T 8eb056b49f4d44869749288abfd50d84 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.472476 25099 ts_tablet_manager.cc:1888] T 5ccc8d73d1ea4539a170022fe6ad890e P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:03.472685 25099 log.cc:1196] T 5ccc8d73d1ea4539a170022fe6ad890e P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/5ccc8d73d1ea4539a170022fe6ad890e
I20240502 01:54:03.473191 25099 ts_tablet_manager.cc:1909] T 5ccc8d73d1ea4539a170022fe6ad890e P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:03.474473 25099 tablet_replica.cc:330] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:03.474768 25099 raft_consensus.cc:2229] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:03.474751 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 5ccc8d73d1ea4539a170022fe6ad890e (table rebalance_test_table_4 [id=0bafdd6983184b96aa892f802d62ebbc]) successfully deleted
I20240502 01:54:03.474747 25230 tablet.cc:1722] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:03.475214 25099 raft_consensus.cc:2258] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.475414 25230 tablet.cc:1620] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:03.477066 25097 ts_tablet_manager.cc:1888] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:03.477257 25097 log.cc:1196] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/8e18fd328b0a45b98eadddb574737672
I20240502 01:54:03.477830 25097 ts_tablet_manager.cc:1909] T 8e18fd328b0a45b98eadddb574737672 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:03.478068 21712 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet f48d8c7e94614d30950e418f2017ac21 (table rebalancer_dynamic_table [id=a71d72fbe2fb496e83394e55a713e713]). Expected version 1 got 0
I20240502 01:54:03.478602 25098 ts_tablet_manager.cc:1875] T 8eb056b49f4d44869749288abfd50d84 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:03.482398 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 8e18fd328b0a45b98eadddb574737672 (table rebalance_test_table_4 [id=0bafdd6983184b96aa892f802d62ebbc]) successfully deleted
I20240502 01:54:03.484463 25100 ts_tablet_manager.cc:1888] T 5ccc8d73d1ea4539a170022fe6ad890e P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:03.484635 25100 log.cc:1196] T 5ccc8d73d1ea4539a170022fe6ad890e P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/5ccc8d73d1ea4539a170022fe6ad890e
I20240502 01:54:03.485194 25100 ts_tablet_manager.cc:1909] T 5ccc8d73d1ea4539a170022fe6ad890e P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:03.486632 25097 tablet_replica.cc:330] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:03.487143 25097 raft_consensus.cc:2229] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:03.487550 25097 raft_consensus.cc:2258] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.488325 25097 ts_tablet_manager.cc:1875] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:03.489154 25099 ts_tablet_manager.cc:1875] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:03.490234 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 5ccc8d73d1ea4539a170022fe6ad890e (table rebalance_test_table_4 [id=0bafdd6983184b96aa892f802d62ebbc]) successfully deleted
I20240502 01:54:03.498637 25098 ts_tablet_manager.cc:1888] T 8eb056b49f4d44869749288abfd50d84 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:03.498800 25098 log.cc:1196] T 8eb056b49f4d44869749288abfd50d84 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/8eb056b49f4d44869749288abfd50d84
I20240502 01:54:03.499284 25098 ts_tablet_manager.cc:1909] T 8eb056b49f4d44869749288abfd50d84 P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:03.500442 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 8eb056b49f4d44869749288abfd50d84 (table rebalance_test_table_4 [id=0bafdd6983184b96aa892f802d62ebbc]) successfully deleted
I20240502 01:54:03.503165 25097 ts_tablet_manager.cc:1888] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:03.503342 25097 log.cc:1196] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/5ccc8d73d1ea4539a170022fe6ad890e
I20240502 01:54:03.503732 25097 ts_tablet_manager.cc:1909] T 5ccc8d73d1ea4539a170022fe6ad890e P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:03.507051 25099 ts_tablet_manager.cc:1888] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:03.507288 25099 log.cc:1196] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/8eb056b49f4d44869749288abfd50d84
I20240502 01:54:03.507860 25099 ts_tablet_manager.cc:1909] T 8eb056b49f4d44869749288abfd50d84 P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:03.508502 25097 tablet_replica.cc:330] T 8eb056b49f4d44869749288abfd50d84 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:03.508684 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 5ccc8d73d1ea4539a170022fe6ad890e (table rebalance_test_table_4 [id=0bafdd6983184b96aa892f802d62ebbc]) successfully deleted
I20240502 01:54:03.509323 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 8eb056b49f4d44869749288abfd50d84 (table rebalance_test_table_4 [id=0bafdd6983184b96aa892f802d62ebbc]) successfully deleted
I20240502 01:54:03.509399 25097 raft_consensus.cc:2229] T 8eb056b49f4d44869749288abfd50d84 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.509649 25097 raft_consensus.cc:2258] T 8eb056b49f4d44869749288abfd50d84 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.510466 25097 ts_tablet_manager.cc:1875] T 8eb056b49f4d44869749288abfd50d84 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:03.515918 25097 ts_tablet_manager.cc:1888] T 8eb056b49f4d44869749288abfd50d84 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:03.516077 25097 log.cc:1196] T 8eb056b49f4d44869749288abfd50d84 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/8eb056b49f4d44869749288abfd50d84
I20240502 01:54:03.516498 25097 ts_tablet_manager.cc:1909] T 8eb056b49f4d44869749288abfd50d84 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:03.517711 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 8eb056b49f4d44869749288abfd50d84 (table rebalance_test_table_4 [id=0bafdd6983184b96aa892f802d62ebbc]) successfully deleted
I20240502 01:54:03.526005 25071 consensus_queue.cc:1035] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:03.532174 25173 consensus_queue.cc:1035] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:03.579869 21707 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:03.615082 25227 tablet.cc:1722] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:03.615649 25227 tablet.cc:1620] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:03.617364 25230 tablet.cc:1722] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:03.617842 25230 tablet.cc:1620] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:03.622236 25228 tablet.cc:1722] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:03.622896 25228 tablet.cc:1620] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:03.633621 21711 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=a71d72fbe2fb496e83394e55a713e713] alter complete (version 2)
I20240502 01:54:03.650067 21707 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:03.661908 22100 tablet_service.cc:1467] Processing CreateTablet for tablet b4225c66fa064750bc2c0f440e71d673 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=06ae89337ff143b5a107c994cbecc25a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:03.662317 24981 tablet_service.cc:1467] Processing CreateTablet for tablet b1c6c622638444bd854616902b8d91ad (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=06ae89337ff143b5a107c994cbecc25a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:03.662662 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4225c66fa064750bc2c0f440e71d673. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.662868 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c6c622638444bd854616902b8d91ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.666172 21836 tablet_service.cc:1467] Processing CreateTablet for tablet b1c6c622638444bd854616902b8d91ad (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=06ae89337ff143b5a107c994cbecc25a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:03.666823 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c6c622638444bd854616902b8d91ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.666898 21837 tablet_service.cc:1467] Processing CreateTablet for tablet 76a262a323dc486f80b91c54436eb8b7 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=06ae89337ff143b5a107c994cbecc25a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:03.667328 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a262a323dc486f80b91c54436eb8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.667989 24980 tablet_service.cc:1467] Processing CreateTablet for tablet 76a262a323dc486f80b91c54436eb8b7 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=06ae89337ff143b5a107c994cbecc25a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:03.668680 24980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a262a323dc486f80b91c54436eb8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.676252 24581 tablet_service.cc:1467] Processing CreateTablet for tablet b4225c66fa064750bc2c0f440e71d673 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=06ae89337ff143b5a107c994cbecc25a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:03.676755 24581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4225c66fa064750bc2c0f440e71d673. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.687317 25261 tablet_bootstrap.cc:492] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:03.689357 24715 tablet_service.cc:1467] Processing CreateTablet for tablet b4225c66fa064750bc2c0f440e71d673 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=06ae89337ff143b5a107c994cbecc25a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:03.689836 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4225c66fa064750bc2c0f440e71d673. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.694236 25261 tablet_bootstrap.cc:654] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.698274 25262 tablet_bootstrap.cc:492] T b4225c66fa064750bc2c0f440e71d673 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:03.701005 25262 tablet_bootstrap.cc:654] T b4225c66fa064750bc2c0f440e71d673 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.702986 25260 tablet_bootstrap.cc:492] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:03.705659 25260 tablet_bootstrap.cc:654] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.707731 24582 tablet_service.cc:1467] Processing CreateTablet for tablet b1c6c622638444bd854616902b8d91ad (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=06ae89337ff143b5a107c994cbecc25a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:03.708367 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c6c622638444bd854616902b8d91ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.708493 25261 tablet_bootstrap.cc:492] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:03.708416 24714 tablet_service.cc:1467] Processing CreateTablet for tablet 76a262a323dc486f80b91c54436eb8b7 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=06ae89337ff143b5a107c994cbecc25a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:03.708662 25261 ts_tablet_manager.cc:1367] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.004s
I20240502 01:54:03.708910 24714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a262a323dc486f80b91c54436eb8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.710418 25261 raft_consensus.cc:348] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:03.710641 25261 raft_consensus.cc:374] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.710740 25261 raft_consensus.cc:729] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.711025 25261 consensus_queue.cc:260] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:03.711745 25261 ts_tablet_manager.cc:1398] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 01:54:03.712428 25261 tablet_bootstrap.cc:492] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:03.712885 25262 tablet_bootstrap.cc:492] T b4225c66fa064750bc2c0f440e71d673 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:03.713083 25262 ts_tablet_manager.cc:1367] T b4225c66fa064750bc2c0f440e71d673 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.005s
I20240502 01:54:03.713882 25262 raft_consensus.cc:348] T b4225c66fa064750bc2c0f440e71d673 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:03.714083 25262 raft_consensus.cc:374] T b4225c66fa064750bc2c0f440e71d673 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.714195 25262 raft_consensus.cc:729] T b4225c66fa064750bc2c0f440e71d673 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.714401 25262 consensus_queue.cc:260] T b4225c66fa064750bc2c0f440e71d673 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:03.714953 25261 tablet_bootstrap.cc:654] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.714993 25262 ts_tablet_manager.cc:1398] T b4225c66fa064750bc2c0f440e71d673 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:03.724848 25260 tablet_bootstrap.cc:492] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:03.725232 25260 ts_tablet_manager.cc:1367] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.006s
I20240502 01:54:03.725694 25263 tablet_bootstrap.cc:492] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:03.726425 25265 tablet_bootstrap.cc:492] T b4225c66fa064750bc2c0f440e71d673 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:03.726771 25260 raft_consensus.cc:348] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:03.729001 25261 tablet_bootstrap.cc:492] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:03.729270 25261 ts_tablet_manager.cc:1367] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20240502 01:54:03.731524 25260 raft_consensus.cc:374] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.731751 25260 raft_consensus.cc:729] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.733408 25261 raft_consensus.cc:348] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:03.734362 25261 raft_consensus.cc:374] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.735356 25261 raft_consensus.cc:729] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.732123 25260 consensus_queue.cc:260] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:03.736838 21715 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:03.736964 25260 ts_tablet_manager.cc:1398] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.012s user 0.001s sys 0.002s
I20240502 01:54:03.738322 25260 tablet_bootstrap.cc:492] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
W20240502 01:54:03.738762 22167 tablet.cc:2369] T b4225c66fa064750bc2c0f440e71d673 P c64840657cdf4b93980bd34120c94d9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:54:03.740161 25260 tablet_bootstrap.cc:654] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.740633 25263 tablet_bootstrap.cc:654] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.742375 25261 consensus_queue.cc:260] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:03.743371 25261 ts_tablet_manager.cc:1398] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 01:54:03.754272 25260 tablet_bootstrap.cc:492] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:03.754583 25260 ts_tablet_manager.cc:1367] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20240502 01:54:03.755661 25260 raft_consensus.cc:348] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:03.756244 25260 raft_consensus.cc:374] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.756387 25260 raft_consensus.cc:729] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.756709 25260 consensus_queue.cc:260] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:03.757398 25260 ts_tablet_manager.cc:1398] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:03.761965 25227 tablet.cc:1722] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:03.765321 25228 tablet.cc:1722] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:03.765839 25230 tablet.cc:1722] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:03.767772 25265 tablet_bootstrap.cc:654] T b4225c66fa064750bc2c0f440e71d673 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.783538 21707 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=a71d72fbe2fb496e83394e55a713e713] alter complete (version 3)
I20240502 01:54:03.790143 25265 tablet_bootstrap.cc:492] T b4225c66fa064750bc2c0f440e71d673 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:03.790390 25265 ts_tablet_manager.cc:1367] T b4225c66fa064750bc2c0f440e71d673 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.064s user 0.015s sys 0.000s
I20240502 01:54:03.791402 25265 raft_consensus.cc:348] T b4225c66fa064750bc2c0f440e71d673 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:03.791723 25265 raft_consensus.cc:374] T b4225c66fa064750bc2c0f440e71d673 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.791968 25265 raft_consensus.cc:729] T b4225c66fa064750bc2c0f440e71d673 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.792255 25265 consensus_queue.cc:260] T b4225c66fa064750bc2c0f440e71d673 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:03.793838 25265 ts_tablet_manager.cc:1398] T b4225c66fa064750bc2c0f440e71d673 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:03.794103 25265 tablet_bootstrap.cc:492] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:03.795840 25265 tablet_bootstrap.cc:654] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.799834 25263 tablet_bootstrap.cc:492] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:03.800133 25263 ts_tablet_manager.cc:1367] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.075s user 0.000s sys 0.007s
I20240502 01:54:03.801316 25263 raft_consensus.cc:348] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:03.801729 25263 raft_consensus.cc:374] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.801895 25263 raft_consensus.cc:729] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.802304 25263 consensus_queue.cc:260] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:03.806747 25263 ts_tablet_manager.cc:1398] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 01:54:03.810197 25263 tablet_bootstrap.cc:492] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
W20240502 01:54:03.810170 21904 tablet.cc:2369] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:54:03.813860 25263 tablet_bootstrap.cc:654] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.826808 21968 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:03.841941 25265 tablet_bootstrap.cc:492] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:03.842171 25265 ts_tablet_manager.cc:1367] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.013s
I20240502 01:54:03.843029 25265 raft_consensus.cc:348] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:03.843289 25265 raft_consensus.cc:374] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.843377 25265 raft_consensus.cc:729] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.843641 25265 consensus_queue.cc:260] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:03.845086 25265 ts_tablet_manager.cc:1398] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:54:03.862357 25263 tablet_bootstrap.cc:492] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:03.862874 25263 ts_tablet_manager.cc:1367] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.006s
I20240502 01:54:03.865696 25263 raft_consensus.cc:348] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:03.866910 25263 raft_consensus.cc:374] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.866978 25263 raft_consensus.cc:729] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.867204 25263 consensus_queue.cc:260] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:03.870615 25263 ts_tablet_manager.cc:1398] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:54:03.889310 21707 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:03.889688 21707 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:03.892997 25222 raft_consensus.cc:482] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:03.893265 25222 raft_consensus.cc:504] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:03.895993 25222 leader_election.cc:288] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:03.896699 21707 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet f48d8c7e94614d30950e418f2017ac21
I20240502 01:54:03.899502 24715 tablet_service.cc:1514] Processing DeleteTablet for tablet f48d8c7e94614d30950e418f2017ac21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:41650
I20240502 01:54:03.900192 24848 tablet_service.cc:1514] Processing DeleteTablet for tablet f48d8c7e94614d30950e418f2017ac21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:51592
I20240502 01:54:03.923153 25116 raft_consensus.cc:482] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:03.923431 25116 raft_consensus.cc:504] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:03.924631 25116 leader_election.cc:288] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:03.951458 21707 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 01:54:03.951995 21707 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 01:54:03.960031 21707 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 21a05c0b257f44708fb0eac9540f2218
I20240502 01:54:03.961371 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 21a05c0b257f44708fb0eac9540f2218 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:03.961830 25097 tablet_replica.cc:330] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:03.961911 21968 tablet_service.cc:1514] Processing DeleteTablet for tablet 21a05c0b257f44708fb0eac9540f2218 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:03.962164 25097 raft_consensus.cc:2229] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:03.962327 25100 tablet_replica.cc:330] T 21a05c0b257f44708fb0eac9540f2218 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:03.962661 25097 raft_consensus.cc:2258] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.962674 25100 raft_consensus.cc:2229] T 21a05c0b257f44708fb0eac9540f2218 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.962994 25100 raft_consensus.cc:2258] T 21a05c0b257f44708fb0eac9540f2218 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.963765 25097 ts_tablet_manager.cc:1875] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:03.964007 25100 ts_tablet_manager.cc:1875] T 21a05c0b257f44708fb0eac9540f2218 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 01:54:03.981284 21924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.195:38879, user_credentials={real_user=slave}} blocked reactor thread for 120841us
I20240502 01:54:03.987625 25290 tablet_replica.cc:330] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:54:03.987998 25290 raft_consensus.cc:2229] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:03.989352 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a262a323dc486f80b91c54436eb8b7" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:03.989756 21857 raft_consensus.cc:2454] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:03.990705 24933 leader_election.cc:302] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:03.991250 25222 raft_consensus.cc:2781] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:03.991384 25222 raft_consensus.cc:482] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:03.991477 25222 raft_consensus.cc:3037] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:03.992416 24981 tablet_service.cc:1514] Processing DeleteTablet for tablet f48d8c7e94614d30950e418f2017ac21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:39964
I20240502 01:54:03.993343 25290 raft_consensus.cc:2258] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.994328 25290 ts_tablet_manager.cc:1875] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:03.994937 25222 raft_consensus.cc:504] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:03.995894 25222 leader_election.cc:288] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:03.996497 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a262a323dc486f80b91c54436eb8b7" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:03.996781 21857 raft_consensus.cc:3037] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.000418 21857 raft_consensus.cc:2454] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:04.000964 24933 leader_election.cc:302] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:04.001543 25222 raft_consensus.cc:2781] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:04.003629 25222 raft_consensus.cc:686] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:54:04.004235 25222 consensus_queue.cc:237] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:04.005388 25291 tablet_replica.cc:330] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08: stopping tablet replica
I20240502 01:54:04.005808 21710 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:04.010769 25174 raft_consensus.cc:482] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:04.011006 25174 raft_consensus.cc:504] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:04.011893 25174 leader_election.cc:288] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:04.015744 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 0689d4851d4143909fe3b7b0d9c7b2d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5ffb0e9de5844f82a2ae0ec1f30c97ef]), partition=
I20240502 01:54:04.016131 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0689d4851d4143909fe3b7b0d9c7b2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.980020 25100 ts_tablet_manager.cc:1888] T 21a05c0b257f44708fb0eac9540f2218 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:04.028988 24918 raft_consensus.cc:482] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:04.029177 24918 raft_consensus.cc:504] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:04.028687 25100 log.cc:1196] T 21a05c0b257f44708fb0eac9540f2218 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/21a05c0b257f44708fb0eac9540f2218
I20240502 01:54:04.030294 25100 ts_tablet_manager.cc:1909] T 21a05c0b257f44708fb0eac9540f2218 P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:04.030325 24918 leader_election.cc:288] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:04.030792 25261 tablet_bootstrap.cc:492] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:04.032868 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c6c622638444bd854616902b8d91ad" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:04.033046 25261 tablet_bootstrap.cc:654] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.033190 25001 raft_consensus.cc:2454] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:54:04.033910 21793 leader_election.cc:302] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:04.035718 24918 raft_consensus.cc:2781] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:04.035887 24918 raft_consensus.cc:482] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:04.035997 24918 raft_consensus.cc:3037] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.042450 24918 raft_consensus.cc:504] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:04.045987 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 21a05c0b257f44708fb0eac9540f2218 (table rebalance_test_table_6 [id=ab3326b89b7f41eab7e16c19f0f25190]) successfully deleted
I20240502 01:54:04.048331 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c6c622638444bd854616902b8d91ad" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:04.048619 25001 raft_consensus.cc:3037] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.051865 25001 raft_consensus.cc:2454] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:54:04.051707 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a262a323dc486f80b91c54436eb8b7" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:04.053025 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4225c66fa064750bc2c0f440e71d673" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:04.053349 24735 raft_consensus.cc:2454] T b4225c66fa064750bc2c0f440e71d673 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:04.053365 21793 leader_election.cc:302] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:04.053984 24785 raft_consensus.cc:2781] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:04.055965 24537 leader_election.cc:302] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:54:04.056442 25116 raft_consensus.cc:2781] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:04.056576 25116 raft_consensus.cc:482] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:04.056663 25116 raft_consensus.cc:3037] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.056998 21707 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 51312aa1f74c43588df6facab76ea1c6
I20240502 01:54:04.058610 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 51312aa1f74c43588df6facab76ea1c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:04.059911 25116 raft_consensus.cc:504] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:04.060781 25116 leader_election.cc:288] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:04.064285 25261 tablet_bootstrap.cc:492] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:04.064474 25261 ts_tablet_manager.cc:1367] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.007s
I20240502 01:54:04.065697 25261 raft_consensus.cc:348] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:04.065819 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4225c66fa064750bc2c0f440e71d673" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:04.065939 25261 raft_consensus.cc:374] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.066037 25261 raft_consensus.cc:729] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.066076 24735 raft_consensus.cc:3037] T b4225c66fa064750bc2c0f440e71d673 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.066736 24918 leader_election.cc:288] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:04.069651 24735 raft_consensus.cc:2454] T b4225c66fa064750bc2c0f440e71d673 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:04.071935 24537 leader_election.cc:302] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:54:04.072084 25291 raft_consensus.cc:2229] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:04.072430 25116 raft_consensus.cc:2781] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:04.072573 21715 catalog_manager.cc:5484] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:04.073185 25261 consensus_queue.cc:260] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:04.074077 25261 ts_tablet_manager.cc:1398] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240502 01:54:04.075001 22229 tablet_service.cc:1514] Processing DeleteTablet for tablet 21a05c0b257f44708fb0eac9540f2218 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:04.078501 24980 tablet_service.cc:1467] Processing CreateTablet for tablet 0689d4851d4143909fe3b7b0d9c7b2d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5ffb0e9de5844f82a2ae0ec1f30c97ef]), partition=
I20240502 01:54:04.078997 24980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0689d4851d4143909fe3b7b0d9c7b2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.083045 22117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4225c66fa064750bc2c0f440e71d673" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:54:04.083140 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4225c66fa064750bc2c0f440e71d673" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:54:04.083384 22117 raft_consensus.cc:2454] T b4225c66fa064750bc2c0f440e71d673 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:04.087749 25260 tablet_bootstrap.cc:492] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:04.088284 25290 ts_tablet_manager.cc:1888] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:04.088439 25290 log.cc:1196] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/f48d8c7e94614d30950e418f2017ac21
I20240502 01:54:04.089701 25260 tablet_bootstrap.cc:654] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.095240 25308 tablet_replica.cc:330] T 21a05c0b257f44708fb0eac9540f2218 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:04.095525 25290 ts_tablet_manager.cc:1909] T f48d8c7e94614d30950e418f2017ac21 P 28157bc3e678428fb5f93c077d528513: Deleting consensus metadata
I20240502 01:54:04.095753 25308 raft_consensus.cc:2229] T 21a05c0b257f44708fb0eac9540f2218 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:04.096187 25308 raft_consensus.cc:2258] T 21a05c0b257f44708fb0eac9540f2218 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.097087 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet f48d8c7e94614d30950e418f2017ac21 (table rebalancer_dynamic_table_new_name [id=a71d72fbe2fb496e83394e55a713e713]) successfully deleted
I20240502 01:54:04.097347 25308 ts_tablet_manager.cc:1875] T 21a05c0b257f44708fb0eac9540f2218 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.102564 25305 tablet_replica.cc:330] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:04.102617 25097 ts_tablet_manager.cc:1888] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:04.102967 25305 raft_consensus.cc:2229] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:04.103464 25305 raft_consensus.cc:2258] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.104398 25305 ts_tablet_manager.cc:1875] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.107234 21707 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 484da49a539a47f199b182041cb659bd
I20240502 01:54:04.109004 25291 raft_consensus.cc:2258] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.109469 21968 tablet_service.cc:1514] Processing DeleteTablet for tablet 484da49a539a47f199b182041cb659bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:04.109970 25100 tablet_replica.cc:330] T 484da49a539a47f199b182041cb659bd P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:04.110265 25100 raft_consensus.cc:2229] T 484da49a539a47f199b182041cb659bd P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:04.110571 25100 raft_consensus.cc:2258] T 484da49a539a47f199b182041cb659bd P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.049055 24848 tablet_service.cc:1467] Processing CreateTablet for tablet 0689d4851d4143909fe3b7b0d9c7b2d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5ffb0e9de5844f82a2ae0ec1f30c97ef]), partition=
I20240502 01:54:04.111200 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0689d4851d4143909fe3b7b0d9c7b2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.111500 25100 ts_tablet_manager.cc:1875] T 484da49a539a47f199b182041cb659bd P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.118422 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet 51312aa1f74c43588df6facab76ea1c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:04.118491 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a262a323dc486f80b91c54436eb8b7" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:04.118769 24735 raft_consensus.cc:2454] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:04.118817 24734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a262a323dc486f80b91c54436eb8b7" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:04.119071 24734 raft_consensus.cc:3037] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.123114 25100 ts_tablet_manager.cc:1888] T 484da49a539a47f199b182041cb659bd P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:04.123302 25100 log.cc:1196] T 484da49a539a47f199b182041cb659bd P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/484da49a539a47f199b182041cb659bd
I20240502 01:54:04.123780 25100 ts_tablet_manager.cc:1909] T 484da49a539a47f199b182041cb659bd P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:04.124042 24734 raft_consensus.cc:2454] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:04.125180 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 484da49a539a47f199b182041cb659bd (table rebalance_test_table_6 [id=ab3326b89b7f41eab7e16c19f0f25190]) successfully deleted
I20240502 01:54:04.126950 25260 tablet_bootstrap.cc:492] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:04.127204 25260 ts_tablet_manager.cc:1367] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.005s
I20240502 01:54:04.128130 25260 raft_consensus.cc:348] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:04.128371 25260 raft_consensus.cc:374] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.128531 25260 raft_consensus.cc:729] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.128808 25260 consensus_queue.cc:260] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:04.130373 25260 ts_tablet_manager.cc:1398] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:04.137639 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet 484da49a539a47f199b182041cb659bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:04.138630 25305 ts_tablet_manager.cc:1888] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:04.138820 25305 log.cc:1196] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/51312aa1f74c43588df6facab76ea1c6
I20240502 01:54:04.139395 25305 ts_tablet_manager.cc:1909] T 51312aa1f74c43588df6facab76ea1c6 P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:04.140901 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 51312aa1f74c43588df6facab76ea1c6 (table rebalance_test_table_6 [id=ab3326b89b7f41eab7e16c19f0f25190]) successfully deleted
I20240502 01:54:04.141547 25308 ts_tablet_manager.cc:1888] T 21a05c0b257f44708fb0eac9540f2218 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:04.141789 25308 log.cc:1196] T 21a05c0b257f44708fb0eac9540f2218 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/21a05c0b257f44708fb0eac9540f2218
I20240502 01:54:04.142432 25308 ts_tablet_manager.cc:1909] T 21a05c0b257f44708fb0eac9540f2218 P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:04.142642 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c6c622638444bd854616902b8d91ad" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:04.143390 24603 raft_consensus.cc:2454] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:54:04.143891 25308 tablet_replica.cc:330] T 51312aa1f74c43588df6facab76ea1c6 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:04.144075 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c6c622638444bd854616902b8d91ad" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:04.144306 24603 raft_consensus.cc:3037] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.144291 25308 raft_consensus.cc:2229] T 51312aa1f74c43588df6facab76ea1c6 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:04.144599 25308 raft_consensus.cc:2258] T 51312aa1f74c43588df6facab76ea1c6 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.145570 25308 ts_tablet_manager.cc:1875] T 51312aa1f74c43588df6facab76ea1c6 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.110742 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 51312aa1f74c43588df6facab76ea1c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:04.110955 21837 tablet_service.cc:1514] Processing DeleteTablet for tablet 484da49a539a47f199b182041cb659bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:04.146623 25097 log.cc:1196] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/21a05c0b257f44708fb0eac9540f2218
I20240502 01:54:04.147146 25097 ts_tablet_manager.cc:1909] T 21a05c0b257f44708fb0eac9540f2218 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:04.147310 24603 raft_consensus.cc:2454] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:54:04.119525 24785 raft_consensus.cc:686] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:54:04.148051 24785 consensus_queue.cc:237] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:04.148408 25097 tablet_replica.cc:330] T 484da49a539a47f199b182041cb659bd P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:04.148650 25097 raft_consensus.cc:2229] T 484da49a539a47f199b182041cb659bd P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:04.149371 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 21a05c0b257f44708fb0eac9540f2218 (table rebalance_test_table_6 [id=ab3326b89b7f41eab7e16c19f0f25190]) successfully deleted
I20240502 01:54:04.150653 25296 tablet_replica.cc:330] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:54:04.150894 25296 raft_consensus.cc:2229] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:04.151248 25296 raft_consensus.cc:2258] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.152465 25296 ts_tablet_manager.cc:1875] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.167500 21715 catalog_manager.cc:5484] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:04.169492 25308 ts_tablet_manager.cc:1888] T 51312aa1f74c43588df6facab76ea1c6 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:04.171766 25291 ts_tablet_manager.cc:1875] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.173154 25308 log.cc:1196] T 51312aa1f74c43588df6facab76ea1c6 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/51312aa1f74c43588df6facab76ea1c6
I20240502 01:54:04.173919 25308 ts_tablet_manager.cc:1909] T 51312aa1f74c43588df6facab76ea1c6 P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:04.176424 25308 tablet_replica.cc:330] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:04.177618 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a262a323dc486f80b91c54436eb8b7" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:04.177999 21857 raft_consensus.cc:2379] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28157bc3e678428fb5f93c077d528513 in current term 1: Already voted for candidate 41c70f373c9d49a49891b03a12c6cb2b in this term.
I20240502 01:54:04.178194 25308 raft_consensus.cc:2229] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:04.178650 25308 raft_consensus.cc:2258] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.179581 24668 leader_election.cc:302] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28157bc3e678428fb5f93c077d528513; no voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77
I20240502 01:54:04.181900 25174 raft_consensus.cc:2726] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:04.186491 25291 ts_tablet_manager.cc:1888] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:04.186648 25116 raft_consensus.cc:686] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:54:04.187019 25291 log.cc:1196] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/wals/f48d8c7e94614d30950e418f2017ac21
I20240502 01:54:04.187511 25308 ts_tablet_manager.cc:1875] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.187764 25291 ts_tablet_manager.cc:1909] T f48d8c7e94614d30950e418f2017ac21 P d7449bb1df214bf4a5f93e4631312f08: Deleting consensus metadata
I20240502 01:54:04.187285 25116 consensus_queue.cc:237] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:04.188616 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 21a05c0b257f44708fb0eac9540f2218 (table rebalance_test_table_6 [id=ab3326b89b7f41eab7e16c19f0f25190]) successfully deleted
I20240502 01:54:04.189383 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 51312aa1f74c43588df6facab76ea1c6 (table rebalance_test_table_6 [id=ab3326b89b7f41eab7e16c19f0f25190]) successfully deleted
I20240502 01:54:04.193270 21700 catalog_manager.cc:4830] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): tablet f48d8c7e94614d30950e418f2017ac21 (table rebalancer_dynamic_table_new_name [id=a71d72fbe2fb496e83394e55a713e713]) successfully deleted
I20240502 01:54:04.193442 25097 raft_consensus.cc:2258] T 484da49a539a47f199b182041cb659bd P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.194383 25097 ts_tablet_manager.cc:1875] T 484da49a539a47f199b182041cb659bd P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.201540 24715 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:04.204404 21836 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:04.208750 25097 ts_tablet_manager.cc:1888] T 484da49a539a47f199b182041cb659bd P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:04.213086 25097 log.cc:1196] T 484da49a539a47f199b182041cb659bd P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/484da49a539a47f199b182041cb659bd
I20240502 01:54:04.213732 25097 ts_tablet_manager.cc:1909] T 484da49a539a47f199b182041cb659bd P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:04.213837 24785 raft_consensus.cc:482] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:04.217486 24785 raft_consensus.cc:504] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:04.219270 25097 tablet_replica.cc:330] T 51312aa1f74c43588df6facab76ea1c6 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:04.219380 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 484da49a539a47f199b182041cb659bd (table rebalance_test_table_6 [id=ab3326b89b7f41eab7e16c19f0f25190]) successfully deleted
I20240502 01:54:04.219659 25097 raft_consensus.cc:2229] T 51312aa1f74c43588df6facab76ea1c6 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:04.220104 25097 raft_consensus.cc:2258] T 51312aa1f74c43588df6facab76ea1c6 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.221025 25097 ts_tablet_manager.cc:1875] T 51312aa1f74c43588df6facab76ea1c6 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.229198 25097 ts_tablet_manager.cc:1888] T 51312aa1f74c43588df6facab76ea1c6 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:04.229437 25097 log.cc:1196] T 51312aa1f74c43588df6facab76ea1c6 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/51312aa1f74c43588df6facab76ea1c6
I20240502 01:54:04.230039 25097 ts_tablet_manager.cc:1909] T 51312aa1f74c43588df6facab76ea1c6 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:04.231388 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 51312aa1f74c43588df6facab76ea1c6 (table rebalance_test_table_6 [id=ab3326b89b7f41eab7e16c19f0f25190]) successfully deleted
I20240502 01:54:04.236173 24981 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:04.237327 24785 leader_election.cc:288] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:04.239440 25222 raft_consensus.cc:482] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:04.239656 25222 raft_consensus.cc:504] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:04.240417 25222 leader_election.cc:288] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:04.241235 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0689d4851d4143909fe3b7b0d9c7b2d7" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:04.241518 21857 raft_consensus.cc:2454] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:04.242054 24933 leader_election.cc:302] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:04.242427 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0689d4851d4143909fe3b7b0d9c7b2d7" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:04.242671 25001 raft_consensus.cc:2454] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:54:04.243139 21793 leader_election.cc:302] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:04.243460 24785 raft_consensus.cc:2781] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:04.243551 24785 raft_consensus.cc:482] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:04.243652 24785 raft_consensus.cc:3037] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.246713 24785 raft_consensus.cc:504] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:04.247475 25222 raft_consensus.cc:2781] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:04.247575 25222 raft_consensus.cc:482] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:04.247682 25222 raft_consensus.cc:3037] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.251309 25316 tablet_bootstrap.cc:492] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:04.253299 25316 tablet_bootstrap.cc:654] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.253291 25222 raft_consensus.cc:504] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:04.254317 25296 ts_tablet_manager.cc:1888] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:04.254457 25296 log.cc:1196] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/f48d8c7e94614d30950e418f2017ac21
I20240502 01:54:04.254539 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0689d4851d4143909fe3b7b0d9c7b2d7" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:04.254848 21857 raft_consensus.cc:2379] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 41c70f373c9d49a49891b03a12c6cb2b in current term 1: Already voted for candidate 759257ea7bef415ca698e60d82199d77 in this term.
I20240502 01:54:04.254995 25296 ts_tablet_manager.cc:1909] T f48d8c7e94614d30950e418f2017ac21 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting consensus metadata
I20240502 01:54:04.257831 25222 leader_election.cc:288] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:04.259037 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet f48d8c7e94614d30950e418f2017ac21 (table rebalancer_dynamic_table_new_name [id=a71d72fbe2fb496e83394e55a713e713]) successfully deleted
I20240502 01:54:04.264041 24785 leader_election.cc:288] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:04.264933 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0689d4851d4143909fe3b7b0d9c7b2d7" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:04.265329 25001 raft_consensus.cc:2379] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 759257ea7bef415ca698e60d82199d77 in current term 1: Already voted for candidate 41c70f373c9d49a49891b03a12c6cb2b in this term.
I20240502 01:54:04.265595 21707 catalog_manager.cc:5484] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:04.268208 24581 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:04.283017 25308 ts_tablet_manager.cc:1888] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:04.283293 25308 log.cc:1196] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/484da49a539a47f199b182041cb659bd
I20240502 01:54:04.283862 25308 ts_tablet_manager.cc:1909] T 484da49a539a47f199b182041cb659bd P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:04.286578 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 484da49a539a47f199b182041cb659bd (table rebalance_test_table_6 [id=ab3326b89b7f41eab7e16c19f0f25190]) successfully deleted
W20240502 01:54:04.300029 25311 log.cc:924] Time spent T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20240502 01:54:04.314503 22231 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:04.339175 22100 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:54:04.341810 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0689d4851d4143909fe3b7b0d9c7b2d7" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:04.344130 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0689d4851d4143909fe3b7b0d9c7b2d7" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
W20240502 01:54:04.344780 24934 leader_election.cc:341] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:54:04.345733 24934 leader_election.cc:341] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:04.345952 24934 leader_election.cc:302] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 759257ea7bef415ca698e60d82199d77, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:04.346518 25222 raft_consensus.cc:2726] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:54:04.348663 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0689d4851d4143909fe3b7b0d9c7b2d7" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:04.349351 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0689d4851d4143909fe3b7b0d9c7b2d7" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
W20240502 01:54:04.349973 21791 leader_election.cc:341] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:54:04.350492 21791 leader_election.cc:341] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:04.350636 21791 leader_election.cc:302] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 41c70f373c9d49a49891b03a12c6cb2b, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:04.351238 24785 raft_consensus.cc:2726] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:54:04.363242 24848 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:04.397730 25316 tablet_bootstrap.cc:492] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:04.398051 25316 ts_tablet_manager.cc:1367] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.147s user 0.004s sys 0.000s
I20240502 01:54:04.399188 25316 raft_consensus.cc:348] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:04.399631 25316 raft_consensus.cc:374] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.399806 25316 raft_consensus.cc:729] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.400295 25316 consensus_queue.cc:260] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:04.406373 25316 ts_tablet_manager.cc:1398] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240502 01:54:04.455695 25222 consensus_queue.cc:1035] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:04.475251 25298 consensus_queue.cc:1035] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:04.535072 25306 consensus_queue.cc:1035] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:04.539983 25304 consensus_queue.cc:1035] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:04.587679 22117 raft_consensus.cc:3037] T b4225c66fa064750bc2c0f440e71d673 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.591943 25116 consensus_queue.cc:1035] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:04.599721 25307 consensus_queue.cc:1035] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:04.610220 21707 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 01:54:04.610559 21707 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 01:54:04.616170 21707 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet e60a8beee9284c98ae6193af2c58e8c7
I20240502 01:54:04.617062 21968 tablet_service.cc:1514] Processing DeleteTablet for tablet e60a8beee9284c98ae6193af2c58e8c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:04 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:04.617421 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet e60a8beee9284c98ae6193af2c58e8c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:04 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:04.617746 25097 tablet_replica.cc:330] T e60a8beee9284c98ae6193af2c58e8c7 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:04.617745 25100 tablet_replica.cc:330] T e60a8beee9284c98ae6193af2c58e8c7 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:04.617982 25100 raft_consensus.cc:2229] T e60a8beee9284c98ae6193af2c58e8c7 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:04.617982 25097 raft_consensus.cc:2229] T e60a8beee9284c98ae6193af2c58e8c7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:04.618222 25097 raft_consensus.cc:2258] T e60a8beee9284c98ae6193af2c58e8c7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.618299 25100 raft_consensus.cc:2258] T e60a8beee9284c98ae6193af2c58e8c7 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.619040 25100 ts_tablet_manager.cc:1875] T e60a8beee9284c98ae6193af2c58e8c7 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.619060 25097 ts_tablet_manager.cc:1875] T e60a8beee9284c98ae6193af2c58e8c7 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.625149 21707 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet ab581c1e0054477c849242f57b5066f0
I20240502 01:54:04.625979 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet ab581c1e0054477c849242f57b5066f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:04 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:04.626137 25097 ts_tablet_manager.cc:1888] T e60a8beee9284c98ae6193af2c58e8c7 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:04.626276 25097 log.cc:1196] T e60a8beee9284c98ae6193af2c58e8c7 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/e60a8beee9284c98ae6193af2c58e8c7
I20240502 01:54:04.626296 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet e60a8beee9284c98ae6193af2c58e8c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:04 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:04.626627 25305 tablet_replica.cc:330] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:04.626766 25097 ts_tablet_manager.cc:1909] T e60a8beee9284c98ae6193af2c58e8c7 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:04.626904 25305 raft_consensus.cc:2229] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:04.627282 25305 raft_consensus.cc:2258] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.627315 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet ab581c1e0054477c849242f57b5066f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:04 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:04.627972 25097 tablet_replica.cc:330] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:04.628034 25305 ts_tablet_manager.cc:1875] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.628013 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet e60a8beee9284c98ae6193af2c58e8c7 (table rebalance_test_table_8 [id=69aebb264a8d4fbebe1a1e02330fb403]) successfully deleted
I20240502 01:54:04.628194 25097 raft_consensus.cc:2229] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:04.628526 25097 raft_consensus.cc:2258] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.627867 21707 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 8421b561a3a2402bafee202c129f4c14
I20240502 01:54:04.629297 25097 ts_tablet_manager.cc:1875] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.629309 25100 ts_tablet_manager.cc:1888] T e60a8beee9284c98ae6193af2c58e8c7 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:04.629431 25100 log.cc:1196] T e60a8beee9284c98ae6193af2c58e8c7 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/e60a8beee9284c98ae6193af2c58e8c7
I20240502 01:54:04.629710 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 8421b561a3a2402bafee202c129f4c14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:04 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:04.630227 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet 8421b561a3a2402bafee202c129f4c14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:04 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:04.631428 25100 ts_tablet_manager.cc:1909] T e60a8beee9284c98ae6193af2c58e8c7 P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:04.632264 21968 tablet_service.cc:1514] Processing DeleteTablet for tablet ab581c1e0054477c849242f57b5066f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:04 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:04.632352 21969 tablet_service.cc:1514] Processing DeleteTablet for tablet 8421b561a3a2402bafee202c129f4c14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:04 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:04.632390 25308 tablet_replica.cc:330] T ab581c1e0054477c849242f57b5066f0 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:04.632624 25308 raft_consensus.cc:2229] T ab581c1e0054477c849242f57b5066f0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:04.632853 25308 raft_consensus.cc:2258] T ab581c1e0054477c849242f57b5066f0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.632839 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet e60a8beee9284c98ae6193af2c58e8c7 (table rebalance_test_table_8 [id=69aebb264a8d4fbebe1a1e02330fb403]) successfully deleted
I20240502 01:54:04.633255 25100 tablet_replica.cc:330] T ab581c1e0054477c849242f57b5066f0 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:04.633487 25100 raft_consensus.cc:2229] T ab581c1e0054477c849242f57b5066f0 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:04.633512 25308 ts_tablet_manager.cc:1875] T ab581c1e0054477c849242f57b5066f0 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.633734 25100 raft_consensus.cc:2258] T ab581c1e0054477c849242f57b5066f0 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.634466 25100 ts_tablet_manager.cc:1875] T ab581c1e0054477c849242f57b5066f0 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.635764 25097 ts_tablet_manager.cc:1888] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:04.635885 25097 log.cc:1196] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/ab581c1e0054477c849242f57b5066f0
I20240502 01:54:04.636334 25097 ts_tablet_manager.cc:1909] T ab581c1e0054477c849242f57b5066f0 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:04.637441 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet ab581c1e0054477c849242f57b5066f0 (table rebalance_test_table_8 [id=69aebb264a8d4fbebe1a1e02330fb403]) successfully deleted
I20240502 01:54:04.639093 25308 ts_tablet_manager.cc:1888] T ab581c1e0054477c849242f57b5066f0 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:04.639250 25305 ts_tablet_manager.cc:1888] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:04.639318 25308 log.cc:1196] T ab581c1e0054477c849242f57b5066f0 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/ab581c1e0054477c849242f57b5066f0
I20240502 01:54:04.639416 25305 log.cc:1196] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/e60a8beee9284c98ae6193af2c58e8c7
I20240502 01:54:04.639750 25308 ts_tablet_manager.cc:1909] T ab581c1e0054477c849242f57b5066f0 P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:04.639850 25305 ts_tablet_manager.cc:1909] T e60a8beee9284c98ae6193af2c58e8c7 P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:04.640049 25100 ts_tablet_manager.cc:1888] T ab581c1e0054477c849242f57b5066f0 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:04.640197 25100 log.cc:1196] T ab581c1e0054477c849242f57b5066f0 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/ab581c1e0054477c849242f57b5066f0
I20240502 01:54:04.640575 25100 ts_tablet_manager.cc:1909] T ab581c1e0054477c849242f57b5066f0 P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:04.640825 25308 tablet_replica.cc:330] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:04.640940 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet ab581c1e0054477c849242f57b5066f0 (table rebalance_test_table_8 [id=69aebb264a8d4fbebe1a1e02330fb403]) successfully deleted
I20240502 01:54:04.641234 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet e60a8beee9284c98ae6193af2c58e8c7 (table rebalance_test_table_8 [id=69aebb264a8d4fbebe1a1e02330fb403]) successfully deleted
I20240502 01:54:04.641233 25308 raft_consensus.cc:2229] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:04.641645 25100 tablet_replica.cc:330] T 8421b561a3a2402bafee202c129f4c14 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:04.641685 25305 tablet_replica.cc:330] T 8421b561a3a2402bafee202c129f4c14 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:04.641724 25308 raft_consensus.cc:2258] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.641741 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet ab581c1e0054477c849242f57b5066f0 (table rebalance_test_table_8 [id=69aebb264a8d4fbebe1a1e02330fb403]) successfully deleted
I20240502 01:54:04.641907 25305 raft_consensus.cc:2229] T 8421b561a3a2402bafee202c129f4c14 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:04.641932 25100 raft_consensus.cc:2229] T 8421b561a3a2402bafee202c129f4c14 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:04.642179 25100 raft_consensus.cc:2258] T 8421b561a3a2402bafee202c129f4c14 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.642216 25305 raft_consensus.cc:2258] T 8421b561a3a2402bafee202c129f4c14 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.642622 25308 ts_tablet_manager.cc:1875] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.642891 25305 ts_tablet_manager.cc:1875] T 8421b561a3a2402bafee202c129f4c14 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.642896 25100 ts_tablet_manager.cc:1875] T 8421b561a3a2402bafee202c129f4c14 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.650980 25308 ts_tablet_manager.cc:1888] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:04.651243 25308 log.cc:1196] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/8421b561a3a2402bafee202c129f4c14
I20240502 01:54:04.651793 25308 ts_tablet_manager.cc:1909] T 8421b561a3a2402bafee202c129f4c14 P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:04.653268 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 8421b561a3a2402bafee202c129f4c14 (table rebalance_test_table_8 [id=69aebb264a8d4fbebe1a1e02330fb403]) successfully deleted
I20240502 01:54:04.653309 25100 ts_tablet_manager.cc:1888] T 8421b561a3a2402bafee202c129f4c14 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:04.653447 25100 log.cc:1196] T 8421b561a3a2402bafee202c129f4c14 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/8421b561a3a2402bafee202c129f4c14
I20240502 01:54:04.653657 25305 ts_tablet_manager.cc:1888] T 8421b561a3a2402bafee202c129f4c14 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:04.653790 25305 log.cc:1196] T 8421b561a3a2402bafee202c129f4c14 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/8421b561a3a2402bafee202c129f4c14
I20240502 01:54:04.653962 25100 ts_tablet_manager.cc:1909] T 8421b561a3a2402bafee202c129f4c14 P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:04.654374 25305 ts_tablet_manager.cc:1909] T 8421b561a3a2402bafee202c129f4c14 P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:04.655318 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 8421b561a3a2402bafee202c129f4c14 (table rebalance_test_table_8 [id=69aebb264a8d4fbebe1a1e02330fb403]) successfully deleted
I20240502 01:54:04.655478 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 8421b561a3a2402bafee202c129f4c14 (table rebalance_test_table_8 [id=69aebb264a8d4fbebe1a1e02330fb403]) successfully deleted
I20240502 01:54:04.762660 25322 raft_consensus.cc:482] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:04.762885 25322 raft_consensus.cc:504] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:04.764467 25322 leader_election.cc:288] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:04.785497 21709 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:04.796046 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0689d4851d4143909fe3b7b0d9c7b2d7" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:04.796115 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0689d4851d4143909fe3b7b0d9c7b2d7" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:04.796376 25001 raft_consensus.cc:2379] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7449bb1df214bf4a5f93e4631312f08 in current term 1: Already voted for candidate 41c70f373c9d49a49891b03a12c6cb2b in this term.
I20240502 01:54:04.796375 21857 raft_consensus.cc:2379] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7449bb1df214bf4a5f93e4631312f08 in current term 1: Already voted for candidate 759257ea7bef415ca698e60d82199d77 in this term.
I20240502 01:54:04.796325 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 5a4a927b8f1740208f99ab1aca81b67a (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c1a2048ad81b49729677603cbe361089]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:04.796790 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a4a927b8f1740208f99ab1aca81b67a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.797331 24581 tablet_service.cc:1467] Processing CreateTablet for tablet 5a4a927b8f1740208f99ab1aca81b67a (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c1a2048ad81b49729677603cbe361089]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:04.797781 24581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a4a927b8f1740208f99ab1aca81b67a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.798166 24981 tablet_service.cc:1467] Processing CreateTablet for tablet 6e1f54cbb35d45cd89ffd1a12eb2ec8b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c1a2048ad81b49729677603cbe361089]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:04.798586 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e1f54cbb35d45cd89ffd1a12eb2ec8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.802448 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 992544bbea6547de96298fd32d74b232 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c1a2048ad81b49729677603cbe361089]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:04.802618 24980 tablet_service.cc:1467] Processing CreateTablet for tablet 992544bbea6547de96298fd32d74b232 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c1a2048ad81b49729677603cbe361089]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:04.802624 24714 tablet_service.cc:1467] Processing CreateTablet for tablet 6e1f54cbb35d45cd89ffd1a12eb2ec8b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c1a2048ad81b49729677603cbe361089]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:04.802935 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 992544bbea6547de96298fd32d74b232. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.802989 24980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 992544bbea6547de96298fd32d74b232. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.803030 24714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e1f54cbb35d45cd89ffd1a12eb2ec8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.819435 24848 tablet_service.cc:1467] Processing CreateTablet for tablet 5a4a927b8f1740208f99ab1aca81b67a (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c1a2048ad81b49729677603cbe361089]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:04.819970 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a4a927b8f1740208f99ab1aca81b67a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.819460 25377 tablet_bootstrap.cc:492] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:04.821403 25375 tablet_bootstrap.cc:492] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:04.822404 25377 tablet_bootstrap.cc:654] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.823864 25375 tablet_bootstrap.cc:654] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.824056 25376 tablet_bootstrap.cc:492] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:04.825834 25376 tablet_bootstrap.cc:654] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.830180 25316 tablet_bootstrap.cc:492] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:04.830619 24801 leader_election.cc:302] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7449bb1df214bf4a5f93e4631312f08; no voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77
I20240502 01:54:04.831804 25322 raft_consensus.cc:3037] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.831914 25316 tablet_bootstrap.cc:654] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.835741 25322 raft_consensus.cc:2726] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:04.835806 24847 tablet_service.cc:1467] Processing CreateTablet for tablet 992544bbea6547de96298fd32d74b232 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c1a2048ad81b49729677603cbe361089]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:04.836349 24847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 992544bbea6547de96298fd32d74b232. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.847793 24848 tablet_service.cc:1467] Processing CreateTablet for tablet 6e1f54cbb35d45cd89ffd1a12eb2ec8b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c1a2048ad81b49729677603cbe361089]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:04.848419 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e1f54cbb35d45cd89ffd1a12eb2ec8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.852717 25382 tablet_bootstrap.cc:492] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:04.858618 25375 tablet_bootstrap.cc:492] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:04.858814 25375 ts_tablet_manager.cc:1367] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.009s
I20240502 01:54:04.859676 25375 raft_consensus.cc:348] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:04.859923 25375 raft_consensus.cc:374] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.860011 25375 raft_consensus.cc:729] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.860311 25376 tablet_bootstrap.cc:492] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:04.860386 25375 consensus_queue.cc:260] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:04.860682 25376 ts_tablet_manager.cc:1367] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.006s
I20240502 01:54:04.861596 25376 raft_consensus.cc:348] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:04.861817 25375 ts_tablet_manager.cc:1398] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240502 01:54:04.862107 25375 tablet_bootstrap.cc:492] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:04.862965 25377 tablet_bootstrap.cc:492] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:04.863188 25377 ts_tablet_manager.cc:1367] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.000s
I20240502 01:54:04.863759 25375 tablet_bootstrap.cc:654] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.865725 25377 raft_consensus.cc:348] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:04.866050 25377 raft_consensus.cc:374] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.866202 25377 raft_consensus.cc:729] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.866505 25377 consensus_queue.cc:260] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:04.869922 25377 ts_tablet_manager.cc:1398] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:04.870878 25316 tablet_bootstrap.cc:492] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:04.871240 25377 tablet_bootstrap.cc:492] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:04.872099 25382 tablet_bootstrap.cc:654] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.872350 25316 ts_tablet_manager.cc:1367] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.042s user 0.001s sys 0.009s
I20240502 01:54:04.873283 25316 raft_consensus.cc:348] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:04.873775 25316 raft_consensus.cc:374] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.873853 25316 raft_consensus.cc:729] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.874053 25376 raft_consensus.cc:374] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.874110 25316 consensus_queue.cc:260] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:04.875315 25376 raft_consensus.cc:729] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.875809 25376 consensus_queue.cc:260] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:04.876762 25377 tablet_bootstrap.cc:654] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.877028 25316 ts_tablet_manager.cc:1398] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:04.880049 25316 tablet_bootstrap.cc:492] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:04.881969 25376 ts_tablet_manager.cc:1398] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240502 01:54:04.883667 25316 tablet_bootstrap.cc:654] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.908241 25382 tablet_bootstrap.cc:492] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:04.908748 25382 ts_tablet_manager.cc:1367] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.000s
I20240502 01:54:04.910928 25382 raft_consensus.cc:348] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:04.911345 25382 raft_consensus.cc:374] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.911587 25382 raft_consensus.cc:729] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.911978 25382 consensus_queue.cc:260] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:04.913897 25382 ts_tablet_manager.cc:1398] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:04.916553 25377 tablet_bootstrap.cc:492] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:04.916800 25377 ts_tablet_manager.cc:1367] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20240502 01:54:04.917871 25377 raft_consensus.cc:348] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:04.919306 25377 raft_consensus.cc:374] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.919396 25377 raft_consensus.cc:729] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.919724 25377 consensus_queue.cc:260] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:04.920291 25377 ts_tablet_manager.cc:1398] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:04.921960 25316 tablet_bootstrap.cc:492] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:04.922740 25375 tablet_bootstrap.cc:492] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:04.922936 25375 ts_tablet_manager.cc:1367] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.061s user 0.020s sys 0.020s
I20240502 01:54:04.922135 25316 ts_tablet_manager.cc:1367] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.020s
I20240502 01:54:04.924300 25316 raft_consensus.cc:348] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:04.925055 25316 raft_consensus.cc:374] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.924922 25375 raft_consensus.cc:348] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:04.925228 25375 raft_consensus.cc:374] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.925235 25316 raft_consensus.cc:729] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.925319 25375 raft_consensus.cc:729] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.925622 25375 consensus_queue.cc:260] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:04.925585 25316 consensus_queue.cc:260] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:04.928198 25316 ts_tablet_manager.cc:1398] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:04.928678 25375 ts_tablet_manager.cc:1398] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240502 01:54:04.929235 25316 tablet_bootstrap.cc:492] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:04.931888 25316 tablet_bootstrap.cc:654] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.937748 25316 tablet_bootstrap.cc:492] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:04.937907 25316 ts_tablet_manager.cc:1367] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240502 01:54:04.938604 25316 raft_consensus.cc:348] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:04.938783 25316 raft_consensus.cc:374] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.938865 25316 raft_consensus.cc:729] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.939131 25316 consensus_queue.cc:260] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:04.939638 25316 ts_tablet_manager.cc:1398] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:05.061340 25222 raft_consensus.cc:482] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:05.061600 25222 raft_consensus.cc:504] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:05.062428 25222 leader_election.cc:288] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:05.063094 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1f54cbb35d45cd89ffd1a12eb2ec8b" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:05.063109 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1f54cbb35d45cd89ffd1a12eb2ec8b" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:05.063376 24735 raft_consensus.cc:2454] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:05.063410 24868 raft_consensus.cc:2454] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:05.063894 24934 leader_election.cc:302] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:05.064172 25222 raft_consensus.cc:2781] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:05.064273 25222 raft_consensus.cc:482] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:05.064385 25222 raft_consensus.cc:3037] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.066566 25222 raft_consensus.cc:504] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:05.067184 25222 leader_election.cc:288] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:05.067668 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1f54cbb35d45cd89ffd1a12eb2ec8b" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:05.067752 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1f54cbb35d45cd89ffd1a12eb2ec8b" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:05.067916 24735 raft_consensus.cc:3037] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.067988 24868 raft_consensus.cc:3037] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.070873 24868 raft_consensus.cc:2454] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:05.070892 24735 raft_consensus.cc:2454] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:05.071319 24934 leader_election.cc:302] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:05.071621 25222 raft_consensus.cc:2781] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:05.072064 25222 raft_consensus.cc:686] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:54:05.072389 25222 consensus_queue.cc:237] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:05.076289 21710 catalog_manager.cc:5484] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:05.131932 21714 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 01:54:05.132138 21714 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 01:54:05.135412 21714 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet dbd6d93ca9224d16a98f11e2e0f1eab0
I20240502 01:54:05.135967 21714 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet c2957ed0b72e4be7bdcb475d6c2e4a54
I20240502 01:54:05.136495 21714 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 904443e236824f1db77d9b4eeeeabaf4
I20240502 01:54:05.136507 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd6d93ca9224d16a98f11e2e0f1eab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:05.136833 25097 tablet_replica.cc:330] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:05.137099 25097 raft_consensus.cc:2229] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:05.137221 21968 tablet_service.cc:1514] Processing DeleteTablet for tablet c2957ed0b72e4be7bdcb475d6c2e4a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:05.137357 21969 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd6d93ca9224d16a98f11e2e0f1eab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:05.137373 25097 raft_consensus.cc:2258] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.137490 25100 tablet_replica.cc:330] T c2957ed0b72e4be7bdcb475d6c2e4a54 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:05.137606 21967 tablet_service.cc:1514] Processing DeleteTablet for tablet 904443e236824f1db77d9b4eeeeabaf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:05.137691 25100 raft_consensus.cc:2229] T c2957ed0b72e4be7bdcb475d6c2e4a54 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:05.137919 25100 raft_consensus.cc:2258] T c2957ed0b72e4be7bdcb475d6c2e4a54 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.138190 25097 ts_tablet_manager.cc:1875] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.138521 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd6d93ca9224d16a98f11e2e0f1eab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:05.138661 25100 ts_tablet_manager.cc:1875] T c2957ed0b72e4be7bdcb475d6c2e4a54 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.138846 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 904443e236824f1db77d9b4eeeeabaf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:05.139878 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet 904443e236824f1db77d9b4eeeeabaf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:05.140462 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet c2957ed0b72e4be7bdcb475d6c2e4a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:05.140801 25305 tablet_replica.cc:330] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:05.141063 25305 raft_consensus.cc:2229] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:05.141309 25305 raft_consensus.cc:2258] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.141670 25308 tablet_replica.cc:330] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:05.141897 25308 raft_consensus.cc:2229] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:05.142020 25305 ts_tablet_manager.cc:1875] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.142261 25308 raft_consensus.cc:2258] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.142994 25308 ts_tablet_manager.cc:1875] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.144973 22229 tablet_service.cc:1514] Processing DeleteTablet for tablet c2957ed0b72e4be7bdcb475d6c2e4a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:05.145123 25097 ts_tablet_manager.cc:1888] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:05.145246 25097 log.cc:1196] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/dbd6d93ca9224d16a98f11e2e0f1eab0
I20240502 01:54:05.145599 25100 ts_tablet_manager.cc:1888] T c2957ed0b72e4be7bdcb475d6c2e4a54 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:05.145727 25097 ts_tablet_manager.cc:1909] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:05.145874 25100 log.cc:1196] T c2957ed0b72e4be7bdcb475d6c2e4a54 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/c2957ed0b72e4be7bdcb475d6c2e4a54
I20240502 01:54:05.146402 25100 ts_tablet_manager.cc:1909] T c2957ed0b72e4be7bdcb475d6c2e4a54 P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:05.146939 25097 tablet_replica.cc:330] T 904443e236824f1db77d9b4eeeeabaf4 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:05.147161 25097 raft_consensus.cc:2229] T 904443e236824f1db77d9b4eeeeabaf4 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:05.147413 25097 raft_consensus.cc:2258] T 904443e236824f1db77d9b4eeeeabaf4 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.147804 25100 tablet_replica.cc:330] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:05.148059 25100 raft_consensus.cc:2229] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:05.148125 25097 ts_tablet_manager.cc:1875] T 904443e236824f1db77d9b4eeeeabaf4 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.148466 25100 raft_consensus.cc:2258] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.149222 25100 ts_tablet_manager.cc:1875] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.150528 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet c2957ed0b72e4be7bdcb475d6c2e4a54 (table rebalance_test_table_10 [id=17fb52b34dea4f27811e830b6350d802]) successfully deleted
I20240502 01:54:05.150821 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet dbd6d93ca9224d16a98f11e2e0f1eab0 (table rebalance_test_table_10 [id=17fb52b34dea4f27811e830b6350d802]) successfully deleted
I20240502 01:54:05.151510 25308 ts_tablet_manager.cc:1888] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:05.151676 25308 log.cc:1196] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/dbd6d93ca9224d16a98f11e2e0f1eab0
I20240502 01:54:05.152091 25308 ts_tablet_manager.cc:1909] T dbd6d93ca9224d16a98f11e2e0f1eab0 P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:05.153143 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet dbd6d93ca9224d16a98f11e2e0f1eab0 (table rebalance_test_table_10 [id=17fb52b34dea4f27811e830b6350d802]) successfully deleted
I20240502 01:54:05.153947 25308 tablet_replica.cc:330] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:05.154726 25308 raft_consensus.cc:2229] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:05.155122 25308 raft_consensus.cc:2258] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.155547 25100 ts_tablet_manager.cc:1888] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:05.155608 25305 ts_tablet_manager.cc:1888] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:05.155787 25100 log.cc:1196] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/dbd6d93ca9224d16a98f11e2e0f1eab0
I20240502 01:54:05.155812 25305 log.cc:1196] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/c2957ed0b72e4be7bdcb475d6c2e4a54
I20240502 01:54:05.155876 25097 ts_tablet_manager.cc:1888] T 904443e236824f1db77d9b4eeeeabaf4 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:05.156009 25097 log.cc:1196] T 904443e236824f1db77d9b4eeeeabaf4 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/904443e236824f1db77d9b4eeeeabaf4
I20240502 01:54:05.156088 25308 ts_tablet_manager.cc:1875] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.156266 25305 ts_tablet_manager.cc:1909] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:05.156375 25100 ts_tablet_manager.cc:1909] T dbd6d93ca9224d16a98f11e2e0f1eab0 P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:05.156448 25097 ts_tablet_manager.cc:1909] T 904443e236824f1db77d9b4eeeeabaf4 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:05.157457 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet c2957ed0b72e4be7bdcb475d6c2e4a54 (table rebalance_test_table_10 [id=17fb52b34dea4f27811e830b6350d802]) successfully deleted
I20240502 01:54:05.157521 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 904443e236824f1db77d9b4eeeeabaf4 (table rebalance_test_table_10 [id=17fb52b34dea4f27811e830b6350d802]) successfully deleted
I20240502 01:54:05.157755 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet dbd6d93ca9224d16a98f11e2e0f1eab0 (table rebalance_test_table_10 [id=17fb52b34dea4f27811e830b6350d802]) successfully deleted
I20240502 01:54:05.157809 25100 tablet_replica.cc:330] T 904443e236824f1db77d9b4eeeeabaf4 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:05.157999 25100 raft_consensus.cc:2229] T 904443e236824f1db77d9b4eeeeabaf4 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:05.158219 25100 raft_consensus.cc:2258] T 904443e236824f1db77d9b4eeeeabaf4 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.158799 25100 ts_tablet_manager.cc:1875] T 904443e236824f1db77d9b4eeeeabaf4 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.161607 25308 ts_tablet_manager.cc:1888] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:05.161728 25308 log.cc:1196] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/904443e236824f1db77d9b4eeeeabaf4
I20240502 01:54:05.162137 25308 ts_tablet_manager.cc:1909] T 904443e236824f1db77d9b4eeeeabaf4 P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:05.163028 25308 tablet_replica.cc:330] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:05.163170 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 904443e236824f1db77d9b4eeeeabaf4 (table rebalance_test_table_10 [id=17fb52b34dea4f27811e830b6350d802]) successfully deleted
I20240502 01:54:05.163262 25308 raft_consensus.cc:2229] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:05.163606 25308 raft_consensus.cc:2258] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.164249 25308 ts_tablet_manager.cc:1875] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.164520 25100 ts_tablet_manager.cc:1888] T 904443e236824f1db77d9b4eeeeabaf4 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:05.164700 25100 log.cc:1196] T 904443e236824f1db77d9b4eeeeabaf4 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/904443e236824f1db77d9b4eeeeabaf4
I20240502 01:54:05.165109 25100 ts_tablet_manager.cc:1909] T 904443e236824f1db77d9b4eeeeabaf4 P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:05.166090 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 904443e236824f1db77d9b4eeeeabaf4 (table rebalance_test_table_10 [id=17fb52b34dea4f27811e830b6350d802]) successfully deleted
I20240502 01:54:05.169204 25308 ts_tablet_manager.cc:1888] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:05.169338 25308 log.cc:1196] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/c2957ed0b72e4be7bdcb475d6c2e4a54
I20240502 01:54:05.169654 25308 ts_tablet_manager.cc:1909] T c2957ed0b72e4be7bdcb475d6c2e4a54 P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:05.170540 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet c2957ed0b72e4be7bdcb475d6c2e4a54 (table rebalance_test_table_10 [id=17fb52b34dea4f27811e830b6350d802]) successfully deleted
I20240502 01:54:05.175379 25222 raft_consensus.cc:482] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:05.175539 25222 raft_consensus.cc:504] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:05.176216 25222 leader_election.cc:288] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:05.176687 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992544bbea6547de96298fd32d74b232" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:05.176748 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992544bbea6547de96298fd32d74b232" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:05.176941 24868 raft_consensus.cc:2454] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:05.176988 21857 raft_consensus.cc:2454] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:05.177351 24934 leader_election.cc:302] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:05.177639 25222 raft_consensus.cc:2781] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:05.177733 25222 raft_consensus.cc:482] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:05.177835 25222 raft_consensus.cc:3037] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.179832 25222 raft_consensus.cc:504] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:05.180388 25222 leader_election.cc:288] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:05.181006 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992544bbea6547de96298fd32d74b232" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:05.181011 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992544bbea6547de96298fd32d74b232" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:05.181226 24868 raft_consensus.cc:3037] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.181227 21857 raft_consensus.cc:3037] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.184146 21857 raft_consensus.cc:2454] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:05.184149 24868 raft_consensus.cc:2454] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:05.184579 24933 leader_election.cc:302] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:05.184931 25222 raft_consensus.cc:2781] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:05.185106 25222 raft_consensus.cc:686] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:54:05.185362 25222 consensus_queue.cc:237] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:05.188072 21714 catalog_manager.cc:5484] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:05.235760 25322 raft_consensus.cc:482] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:05.235978 25322 raft_consensus.cc:504] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:05.236598 25322 leader_election.cc:288] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:05.244400 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a4a927b8f1740208f99ab1aca81b67a" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:05.244705 24735 raft_consensus.cc:2454] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:05.244696 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a4a927b8f1740208f99ab1aca81b67a" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:05.244959 24603 raft_consensus.cc:2454] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:05.245554 24802 leader_election.cc:302] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:05.246031 25322 raft_consensus.cc:2781] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:05.246134 25322 raft_consensus.cc:482] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:05.246268 25322 raft_consensus.cc:3037] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.248628 25322 raft_consensus.cc:504] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:05.249433 25322 leader_election.cc:288] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:05.250386 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a4a927b8f1740208f99ab1aca81b67a" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:05.250607 24603 raft_consensus.cc:3037] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.253834 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a4a927b8f1740208f99ab1aca81b67a" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:05.254073 24735 raft_consensus.cc:3037] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.256812 24735 raft_consensus.cc:2454] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:05.257555 24603 raft_consensus.cc:2454] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:05.258136 24802 leader_election.cc:302] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:05.258420 25322 raft_consensus.cc:2781] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:05.259956 25322 raft_consensus.cc:686] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:05.260536 25322 consensus_queue.cc:237] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:05.280473 21710 catalog_manager.cc:5484] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:05.567281 25222 consensus_queue.cc:1035] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:05.572851 25222 consensus_queue.cc:1035] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:05.580325 25222 consensus_queue.cc:1035] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:05.586035 25389 consensus_queue.cc:1035] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:05.640413 21714 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 01:54:05.640678 21714 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 01:54:05.644346 21714 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 913d652575644bd4a8d25ba085e1715e
I20240502 01:54:05.645165 21714 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 26138bfc68834edaa4dcda8edcb5d69c
I20240502 01:54:05.645397 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 913d652575644bd4a8d25ba085e1715e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:05.645700 25097 tablet_replica.cc:330] T 913d652575644bd4a8d25ba085e1715e P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:05.645897 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 913d652575644bd4a8d25ba085e1715e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:05.645972 25097 raft_consensus.cc:2229] T 913d652575644bd4a8d25ba085e1715e P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:05.646114 21967 tablet_service.cc:1514] Processing DeleteTablet for tablet 913d652575644bd4a8d25ba085e1715e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:05.646260 25097 raft_consensus.cc:2258] T 913d652575644bd4a8d25ba085e1715e P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.646468 25305 tablet_replica.cc:330] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:05.646466 25100 tablet_replica.cc:330] T 913d652575644bd4a8d25ba085e1715e P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:05.646672 25305 raft_consensus.cc:2229] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:05.646687 25100 raft_consensus.cc:2229] T 913d652575644bd4a8d25ba085e1715e P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:05.646960 25100 raft_consensus.cc:2258] T 913d652575644bd4a8d25ba085e1715e P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.646989 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 26138bfc68834edaa4dcda8edcb5d69c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:05.647037 25305 raft_consensus.cc:2258] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.647182 25097 ts_tablet_manager.cc:1875] T 913d652575644bd4a8d25ba085e1715e P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.647221 21714 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet c3acd37f59844c0783643e9aef41ea9c
I20240502 01:54:05.647719 25305 ts_tablet_manager.cc:1875] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.647719 25100 ts_tablet_manager.cc:1875] T 913d652575644bd4a8d25ba085e1715e P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.649194 21969 tablet_service.cc:1514] Processing DeleteTablet for tablet c3acd37f59844c0783643e9aef41ea9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:05.649756 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet c3acd37f59844c0783643e9aef41ea9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:05.650331 25322 consensus_queue.cc:1035] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:05.651428 21967 tablet_service.cc:1514] Processing DeleteTablet for tablet 26138bfc68834edaa4dcda8edcb5d69c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:05.652097 22229 tablet_service.cc:1514] Processing DeleteTablet for tablet 26138bfc68834edaa4dcda8edcb5d69c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:05.652380 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet c3acd37f59844c0783643e9aef41ea9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:05.652622 25308 tablet_replica.cc:330] T 26138bfc68834edaa4dcda8edcb5d69c P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:05.652851 25308 raft_consensus.cc:2229] T 26138bfc68834edaa4dcda8edcb5d69c P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:05.653163 25308 raft_consensus.cc:2258] T 26138bfc68834edaa4dcda8edcb5d69c P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.653918 25308 ts_tablet_manager.cc:1875] T 26138bfc68834edaa4dcda8edcb5d69c P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.654366 25097 ts_tablet_manager.cc:1888] T 913d652575644bd4a8d25ba085e1715e P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:05.654568 25097 log.cc:1196] T 913d652575644bd4a8d25ba085e1715e P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/913d652575644bd4a8d25ba085e1715e
I20240502 01:54:05.655025 25097 ts_tablet_manager.cc:1909] T 913d652575644bd4a8d25ba085e1715e P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:05.655887 25305 ts_tablet_manager.cc:1888] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:05.656083 25305 log.cc:1196] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/913d652575644bd4a8d25ba085e1715e
I20240502 01:54:05.656162 25097 tablet_replica.cc:330] T 26138bfc68834edaa4dcda8edcb5d69c P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:05.656400 25097 raft_consensus.cc:2229] T 26138bfc68834edaa4dcda8edcb5d69c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:05.656543 25100 ts_tablet_manager.cc:1888] T 913d652575644bd4a8d25ba085e1715e P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:05.656658 25097 raft_consensus.cc:2258] T 26138bfc68834edaa4dcda8edcb5d69c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.656708 25100 log.cc:1196] T 913d652575644bd4a8d25ba085e1715e P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/913d652575644bd4a8d25ba085e1715e
I20240502 01:54:05.656847 25305 ts_tablet_manager.cc:1909] T 913d652575644bd4a8d25ba085e1715e P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:05.657521 25097 ts_tablet_manager.cc:1875] T 26138bfc68834edaa4dcda8edcb5d69c P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.658457 25305 tablet_replica.cc:330] T c3acd37f59844c0783643e9aef41ea9c P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:05.658759 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 913d652575644bd4a8d25ba085e1715e (table rebalance_test_table_12 [id=906b29d0bc414f019de158cfa649177d]) successfully deleted
I20240502 01:54:05.658808 25305 raft_consensus.cc:2229] T c3acd37f59844c0783643e9aef41ea9c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:05.659190 25305 raft_consensus.cc:2258] T c3acd37f59844c0783643e9aef41ea9c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.659368 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 913d652575644bd4a8d25ba085e1715e (table rebalance_test_table_12 [id=906b29d0bc414f019de158cfa649177d]) successfully deleted
I20240502 01:54:05.660279 25305 ts_tablet_manager.cc:1875] T c3acd37f59844c0783643e9aef41ea9c P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.661617 25100 ts_tablet_manager.cc:1909] T 913d652575644bd4a8d25ba085e1715e P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:05.662459 25308 ts_tablet_manager.cc:1888] T 26138bfc68834edaa4dcda8edcb5d69c P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:05.662614 25308 log.cc:1196] T 26138bfc68834edaa4dcda8edcb5d69c P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/26138bfc68834edaa4dcda8edcb5d69c
I20240502 01:54:05.663277 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 913d652575644bd4a8d25ba085e1715e (table rebalance_test_table_12 [id=906b29d0bc414f019de158cfa649177d]) successfully deleted
I20240502 01:54:05.663502 25308 ts_tablet_manager.cc:1909] T 26138bfc68834edaa4dcda8edcb5d69c P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:05.663935 25100 tablet_replica.cc:330] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:05.664181 25100 raft_consensus.cc:2229] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:05.664729 25100 raft_consensus.cc:2258] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.664774 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 26138bfc68834edaa4dcda8edcb5d69c (table rebalance_test_table_12 [id=906b29d0bc414f019de158cfa649177d]) successfully deleted
I20240502 01:54:05.665161 25308 tablet_replica.cc:330] T c3acd37f59844c0783643e9aef41ea9c P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:05.665409 25308 raft_consensus.cc:2229] T c3acd37f59844c0783643e9aef41ea9c P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:05.665531 25100 ts_tablet_manager.cc:1875] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.665661 25308 raft_consensus.cc:2258] T c3acd37f59844c0783643e9aef41ea9c P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.666493 25308 ts_tablet_manager.cc:1875] T c3acd37f59844c0783643e9aef41ea9c P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.670419 25305 ts_tablet_manager.cc:1888] T c3acd37f59844c0783643e9aef41ea9c P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:05.670709 25305 log.cc:1196] T c3acd37f59844c0783643e9aef41ea9c P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/c3acd37f59844c0783643e9aef41ea9c
I20240502 01:54:05.671216 25305 ts_tablet_manager.cc:1909] T c3acd37f59844c0783643e9aef41ea9c P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:05.671661 25097 ts_tablet_manager.cc:1888] T 26138bfc68834edaa4dcda8edcb5d69c P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:05.671816 25097 log.cc:1196] T 26138bfc68834edaa4dcda8edcb5d69c P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/26138bfc68834edaa4dcda8edcb5d69c
I20240502 01:54:05.672303 25097 ts_tablet_manager.cc:1909] T 26138bfc68834edaa4dcda8edcb5d69c P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:05.672665 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet c3acd37f59844c0783643e9aef41ea9c (table rebalance_test_table_12 [id=906b29d0bc414f019de158cfa649177d]) successfully deleted
I20240502 01:54:05.673642 25308 ts_tablet_manager.cc:1888] T c3acd37f59844c0783643e9aef41ea9c P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:05.673890 25308 log.cc:1196] T c3acd37f59844c0783643e9aef41ea9c P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/c3acd37f59844c0783643e9aef41ea9c
I20240502 01:54:05.674335 25308 ts_tablet_manager.cc:1909] T c3acd37f59844c0783643e9aef41ea9c P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:05.675458 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet c3acd37f59844c0783643e9aef41ea9c (table rebalance_test_table_12 [id=906b29d0bc414f019de158cfa649177d]) successfully deleted
I20240502 01:54:05.677212 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 26138bfc68834edaa4dcda8edcb5d69c (table rebalance_test_table_12 [id=906b29d0bc414f019de158cfa649177d]) successfully deleted
I20240502 01:54:05.678172 25394 consensus_queue.cc:1035] T 5a4a927b8f1740208f99ab1aca81b67a P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:05.679113 25100 ts_tablet_manager.cc:1888] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:05.679378 25100 log.cc:1196] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/c3acd37f59844c0783643e9aef41ea9c
I20240502 01:54:05.679919 25100 ts_tablet_manager.cc:1909] T c3acd37f59844c0783643e9aef41ea9c P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:05.681058 25100 tablet_replica.cc:330] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:05.681123 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet c3acd37f59844c0783643e9aef41ea9c (table rebalance_test_table_12 [id=906b29d0bc414f019de158cfa649177d]) successfully deleted
I20240502 01:54:05.681320 25100 raft_consensus.cc:2229] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:05.681692 25100 raft_consensus.cc:2258] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.682868 25100 ts_tablet_manager.cc:1875] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.688079 25100 ts_tablet_manager.cc:1888] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:05.688205 25100 log.cc:1196] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/26138bfc68834edaa4dcda8edcb5d69c
I20240502 01:54:05.688542 25100 ts_tablet_manager.cc:1909] T 26138bfc68834edaa4dcda8edcb5d69c P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:05.689451 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 26138bfc68834edaa4dcda8edcb5d69c (table rebalance_test_table_12 [id=906b29d0bc414f019de158cfa649177d]) successfully deleted
I20240502 01:54:05.882191 25304 raft_consensus.cc:482] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:05.882385 25304 raft_consensus.cc:504] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:05.883020 25304 leader_election.cc:288] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:05.883527 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0689d4851d4143909fe3b7b0d9c7b2d7" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:05.883625 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0689d4851d4143909fe3b7b0d9c7b2d7" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:05.883777 24868 raft_consensus.cc:2454] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:54:05.883899 25001 raft_consensus.cc:2454] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:54:05.884256 21791 leader_election.cc:302] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:05.884555 25304 raft_consensus.cc:2781] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:05.884652 25304 raft_consensus.cc:482] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:05.884760 25304 raft_consensus.cc:3037] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:05.888020 25304 raft_consensus.cc:504] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:05.888784 25304 leader_election.cc:288] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:05.889243 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0689d4851d4143909fe3b7b0d9c7b2d7" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:05.889281 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0689d4851d4143909fe3b7b0d9c7b2d7" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:05.889498 24868 raft_consensus.cc:3037] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:05.889514 25001 raft_consensus.cc:3037] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:05.892364 24868 raft_consensus.cc:2454] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 2.
I20240502 01:54:05.892844 21791 leader_election.cc:302] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:05.893324 25001 raft_consensus.cc:2454] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 2.
I20240502 01:54:05.894779 25409 raft_consensus.cc:2781] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:05.895491 21714 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:05.895056 25409 raft_consensus.cc:686] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 2 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:54:05.899041 25409 consensus_queue.cc:237] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:05.908223 24848 tablet_service.cc:1467] Processing CreateTablet for tablet 9f05784e69ff49c8bed9720812a067a3 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=16e87707ee294e7c9eed52900510d340]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:05.908758 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f05784e69ff49c8bed9720812a067a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:05.909375 24847 tablet_service.cc:1467] Processing CreateTablet for tablet 2362b0bec07540f1bc0efedbcf61599c (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=16e87707ee294e7c9eed52900510d340]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:05.909966 24847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2362b0bec07540f1bc0efedbcf61599c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:05.915489 21967 tablet_service.cc:1467] Processing CreateTablet for tablet 9f05784e69ff49c8bed9720812a067a3 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=16e87707ee294e7c9eed52900510d340]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:05.916016 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f05784e69ff49c8bed9720812a067a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:05.916245 21714 catalog_manager.cc:5484] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 2, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 2 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:05.917191 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 9f05784e69ff49c8bed9720812a067a3 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=16e87707ee294e7c9eed52900510d340]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:05.917657 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f05784e69ff49c8bed9720812a067a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:05.918584 24581 tablet_service.cc:1467] Processing CreateTablet for tablet 2362b0bec07540f1bc0efedbcf61599c (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=16e87707ee294e7c9eed52900510d340]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:05.919036 24581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2362b0bec07540f1bc0efedbcf61599c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:05.921156 24582 tablet_service.cc:1467] Processing CreateTablet for tablet b60b948dae4841dd9c0919f2fcac5299 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=16e87707ee294e7c9eed52900510d340]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:05.921576 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60b948dae4841dd9c0919f2fcac5299. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:05.922168 24714 tablet_service.cc:1467] Processing CreateTablet for tablet b60b948dae4841dd9c0919f2fcac5299 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=16e87707ee294e7c9eed52900510d340]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:05.922624 24714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60b948dae4841dd9c0919f2fcac5299. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:05.923377 24846 tablet_service.cc:1467] Processing CreateTablet for tablet b60b948dae4841dd9c0919f2fcac5299 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=16e87707ee294e7c9eed52900510d340]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:05.923799 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60b948dae4841dd9c0919f2fcac5299. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:05.923735 24980 tablet_service.cc:1467] Processing CreateTablet for tablet 2362b0bec07540f1bc0efedbcf61599c (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=16e87707ee294e7c9eed52900510d340]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:05.924175 24980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2362b0bec07540f1bc0efedbcf61599c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:05.940286 25418 tablet_bootstrap.cc:492] T 9f05784e69ff49c8bed9720812a067a3 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:05.940809 25414 tablet_bootstrap.cc:492] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:05.942590 25412 tablet_bootstrap.cc:492] T 2362b0bec07540f1bc0efedbcf61599c P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:05.944376 25418 tablet_bootstrap.cc:654] T 9f05784e69ff49c8bed9720812a067a3 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:05.944384 25412 tablet_bootstrap.cc:654] T 2362b0bec07540f1bc0efedbcf61599c P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:05.945274 25419 tablet_bootstrap.cc:492] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:05.946280 25414 tablet_bootstrap.cc:654] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:05.947027 25419 tablet_bootstrap.cc:654] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:05.952205 25414 tablet_bootstrap.cc:492] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:05.952507 25414 ts_tablet_manager.cc:1367] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20240502 01:54:05.955339 25414 raft_consensus.cc:348] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:05.955833 25414 raft_consensus.cc:374] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:05.956007 25414 raft_consensus.cc:729] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:05.956379 25414 consensus_queue.cc:260] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:05.954901 25417 tablet_bootstrap.cc:492] T 2362b0bec07540f1bc0efedbcf61599c P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:05.965492 25412 tablet_bootstrap.cc:492] T 2362b0bec07540f1bc0efedbcf61599c P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:05.965703 25412 ts_tablet_manager.cc:1367] T 2362b0bec07540f1bc0efedbcf61599c P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.004s
I20240502 01:54:05.966341 25416 tablet_bootstrap.cc:492] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:05.966712 25417 tablet_bootstrap.cc:654] T 2362b0bec07540f1bc0efedbcf61599c P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:05.966563 25412 raft_consensus.cc:348] T 2362b0bec07540f1bc0efedbcf61599c P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:05.966804 25412 raft_consensus.cc:374] T 2362b0bec07540f1bc0efedbcf61599c P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:05.966904 25412 raft_consensus.cc:729] T 2362b0bec07540f1bc0efedbcf61599c P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:05.967207 25412 consensus_queue.cc:260] T 2362b0bec07540f1bc0efedbcf61599c P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:05.968638 25412 ts_tablet_manager.cc:1398] T 2362b0bec07540f1bc0efedbcf61599c P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 01:54:05.968972 25412 tablet_bootstrap.cc:492] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:05.970644 25412 tablet_bootstrap.cc:654] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:05.970690 25414 ts_tablet_manager.cc:1398] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240502 01:54:05.977311 25418 tablet_bootstrap.cc:492] T 9f05784e69ff49c8bed9720812a067a3 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:05.977422 25419 tablet_bootstrap.cc:492] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:05.977617 25419 ts_tablet_manager.cc:1367] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.009s
I20240502 01:54:05.978494 25419 raft_consensus.cc:348] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:05.978884 25419 raft_consensus.cc:374] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:05.979039 25419 raft_consensus.cc:729] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:05.979409 25419 consensus_queue.cc:260] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:05.979573 25416 tablet_bootstrap.cc:654] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:05.981076 25419 ts_tablet_manager.cc:1398] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:54:05.981456 25419 tablet_bootstrap.cc:492] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:05.984238 25419 tablet_bootstrap.cc:654] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:05.988126 25418 ts_tablet_manager.cc:1367] T 9f05784e69ff49c8bed9720812a067a3 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.022s
I20240502 01:54:05.991926 25419 tablet_bootstrap.cc:492] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:05.991119 25418 raft_consensus.cc:348] T 9f05784e69ff49c8bed9720812a067a3 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:05.992246 25418 raft_consensus.cc:374] T 9f05784e69ff49c8bed9720812a067a3 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:05.992357 25418 raft_consensus.cc:729] T 9f05784e69ff49c8bed9720812a067a3 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:05.993147 25419 ts_tablet_manager.cc:1367] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240502 01:54:05.994364 25419 raft_consensus.cc:348] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:05.994612 25418 consensus_queue.cc:260] T 9f05784e69ff49c8bed9720812a067a3 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:05.996138 25419 raft_consensus.cc:374] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:05.996467 25419 raft_consensus.cc:729] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:05.996935 25419 consensus_queue.cc:260] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:05.998760 25418 ts_tablet_manager.cc:1398] T 9f05784e69ff49c8bed9720812a067a3 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20240502 01:54:05.999260 25419 ts_tablet_manager.cc:1398] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:54:06.002954 25412 tablet_bootstrap.cc:492] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:06.003217 25412 ts_tablet_manager.cc:1367] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.002s
I20240502 01:54:06.004135 25412 raft_consensus.cc:348] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:06.004467 25412 raft_consensus.cc:374] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:06.004608 25412 raft_consensus.cc:729] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:06.005030 25412 consensus_queue.cc:260] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:06.006985 25412 ts_tablet_manager.cc:1398] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:06.007397 25412 tablet_bootstrap.cc:492] T 9f05784e69ff49c8bed9720812a067a3 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:06.009429 25412 tablet_bootstrap.cc:654] T 9f05784e69ff49c8bed9720812a067a3 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:06.012490 25417 tablet_bootstrap.cc:492] T 2362b0bec07540f1bc0efedbcf61599c P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:06.014829 25417 ts_tablet_manager.cc:1367] T 2362b0bec07540f1bc0efedbcf61599c P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.060s user 0.008s sys 0.000s
I20240502 01:54:06.016706 25417 raft_consensus.cc:348] T 2362b0bec07540f1bc0efedbcf61599c P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:06.019896 25417 raft_consensus.cc:374] T 2362b0bec07540f1bc0efedbcf61599c P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:06.020161 25322 raft_consensus.cc:482] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:06.020267 25417 raft_consensus.cc:729] T 2362b0bec07540f1bc0efedbcf61599c P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:06.020359 25322 raft_consensus.cc:504] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:06.020553 25417 consensus_queue.cc:260] T 2362b0bec07540f1bc0efedbcf61599c P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:06.021169 25322 leader_election.cc:288] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:06.022393 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60b948dae4841dd9c0919f2fcac5299" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:06.022709 24735 raft_consensus.cc:2454] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:06.023069 25416 tablet_bootstrap.cc:492] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:06.023265 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60b948dae4841dd9c0919f2fcac5299" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:06.023550 24603 raft_consensus.cc:2454] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:06.024231 24802 leader_election.cc:302] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:06.024917 25416 ts_tablet_manager.cc:1367] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.000s
I20240502 01:54:06.024920 25322 raft_consensus.cc:2781] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:06.025035 25322 raft_consensus.cc:482] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:06.025161 25322 raft_consensus.cc:3037] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.026254 25417 ts_tablet_manager.cc:1398] T 2362b0bec07540f1bc0efedbcf61599c P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:54:06.028039 25322 raft_consensus.cc:504] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:06.028358 25416 raft_consensus.cc:348] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:06.028604 25416 raft_consensus.cc:374] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:06.028702 25416 raft_consensus.cc:729] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:06.029418 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60b948dae4841dd9c0919f2fcac5299" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:06.029697 24603 raft_consensus.cc:3037] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.029659 25416 consensus_queue.cc:260] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:06.033852 25416 ts_tablet_manager.cc:1398] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240502 01:54:06.035113 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60b948dae4841dd9c0919f2fcac5299" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:06.035346 24735 raft_consensus.cc:3037] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.039005 24735 raft_consensus.cc:2454] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:06.039709 24802 leader_election.cc:302] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:06.040143 25394 raft_consensus.cc:2781] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
W20240502 01:54:06.040588 22036 tablet.cc:2369] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:54:06.050042 24603 raft_consensus.cc:2454] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:06.052428 21967 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:06.061321 25394 raft_consensus.cc:686] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:06.061817 25394 consensus_queue.cc:237] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:06.065124 25322 leader_election.cc:288] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:06.073701 25412 tablet_bootstrap.cc:492] T 9f05784e69ff49c8bed9720812a067a3 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:06.073940 25412 ts_tablet_manager.cc:1367] T 9f05784e69ff49c8bed9720812a067a3 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.000s
I20240502 01:54:06.075106 25412 raft_consensus.cc:348] T 9f05784e69ff49c8bed9720812a067a3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:06.075415 25412 raft_consensus.cc:374] T 9f05784e69ff49c8bed9720812a067a3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:06.075516 25412 raft_consensus.cc:729] T 9f05784e69ff49c8bed9720812a067a3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:06.075857 25412 consensus_queue.cc:260] T 9f05784e69ff49c8bed9720812a067a3 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:06.076669 25412 ts_tablet_manager.cc:1398] T 9f05784e69ff49c8bed9720812a067a3 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:06.078239 21714 catalog_manager.cc:5484] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:06.098454 24714 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:06.107095 21836 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:06.112743 24581 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:06.121223 22100 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:06.122687 24848 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:06.136673 22231 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:06.151793 21714 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 01:54:06.152065 21714 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 01:54:06.158481 24980 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:54:06.171717 21714 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet c6fa6da27fc745e79963ffa294472d9e
I20240502 01:54:06.172626 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet c6fa6da27fc745e79963ffa294472d9e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:06.173071 25308 tablet_replica.cc:330] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:06.173312 25308 raft_consensus.cc:2229] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:06.173720 25308 raft_consensus.cc:2258] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.174566 25308 ts_tablet_manager.cc:1875] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.175192 25307 raft_consensus.cc:482] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:06.175390 25307 raft_consensus.cc:504] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:06.175590 21967 tablet_service.cc:1514] Processing DeleteTablet for tablet c6fa6da27fc745e79963ffa294472d9e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:06.175917 25100 tablet_replica.cc:330] T c6fa6da27fc745e79963ffa294472d9e P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:06.176187 25100 raft_consensus.cc:2229] T c6fa6da27fc745e79963ffa294472d9e P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:06.176332 25307 leader_election.cc:288] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:06.176499 25100 raft_consensus.cc:2258] T c6fa6da27fc745e79963ffa294472d9e P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.177028 21714 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 68e1096b01964772ac02755f1ae21c53
I20240502 01:54:06.177206 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet c6fa6da27fc745e79963ffa294472d9e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:06.177564 25100 ts_tablet_manager.cc:1875] T c6fa6da27fc745e79963ffa294472d9e P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.177924 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 68e1096b01964772ac02755f1ae21c53 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:06.178702 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2362b0bec07540f1bc0efedbcf61599c" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:06.178953 24868 raft_consensus.cc:2454] T 2362b0bec07540f1bc0efedbcf61599c P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:06.179454 21714 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 270e238486f24381815427c4ee2d587a
I20240502 01:54:06.181097 25097 tablet_replica.cc:330] T c6fa6da27fc745e79963ffa294472d9e P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:06.181134 24537 leader_election.cc:302] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:06.181535 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet 68e1096b01964772ac02755f1ae21c53 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:06.182247 21969 tablet_service.cc:1514] Processing DeleteTablet for tablet 270e238486f24381815427c4ee2d587a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:06.182830 21967 tablet_service.cc:1514] Processing DeleteTablet for tablet 68e1096b01964772ac02755f1ae21c53 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:06.183043 25097 raft_consensus.cc:2229] T c6fa6da27fc745e79963ffa294472d9e P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:06.183374 25097 raft_consensus.cc:2258] T c6fa6da27fc745e79963ffa294472d9e P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.184151 25097 ts_tablet_manager.cc:1875] T c6fa6da27fc745e79963ffa294472d9e P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.187386 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 270e238486f24381815427c4ee2d587a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:06.187706 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet 270e238486f24381815427c4ee2d587a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:06.188061 25100 ts_tablet_manager.cc:1888] T c6fa6da27fc745e79963ffa294472d9e P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:06.191417 25100 log.cc:1196] T c6fa6da27fc745e79963ffa294472d9e P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/c6fa6da27fc745e79963ffa294472d9e
I20240502 01:54:06.191917 25100 ts_tablet_manager.cc:1909] T c6fa6da27fc745e79963ffa294472d9e P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:06.193039 25097 ts_tablet_manager.cc:1888] T c6fa6da27fc745e79963ffa294472d9e P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:06.193204 25097 log.cc:1196] T c6fa6da27fc745e79963ffa294472d9e P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/c6fa6da27fc745e79963ffa294472d9e
I20240502 01:54:06.193619 25097 ts_tablet_manager.cc:1909] T c6fa6da27fc745e79963ffa294472d9e P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:06.194454 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet c6fa6da27fc745e79963ffa294472d9e (table rebalance_test_table_14 [id=27c9bf24f886426b92b80b01e4a8d669]) successfully deleted
I20240502 01:54:06.197094 25100 tablet_replica.cc:330] T 270e238486f24381815427c4ee2d587a P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:06.197316 25100 raft_consensus.cc:2229] T 270e238486f24381815427c4ee2d587a P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:06.197573 25100 raft_consensus.cc:2258] T 270e238486f24381815427c4ee2d587a P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.198395 25100 ts_tablet_manager.cc:1875] T 270e238486f24381815427c4ee2d587a P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.199304 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet c6fa6da27fc745e79963ffa294472d9e (table rebalance_test_table_14 [id=27c9bf24f886426b92b80b01e4a8d669]) successfully deleted
I20240502 01:54:06.204198 25097 tablet_replica.cc:330] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:06.204478 25097 raft_consensus.cc:2229] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:06.205902 25097 raft_consensus.cc:2258] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.206904 25452 tablet_replica.cc:330] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:06.207785 25308 ts_tablet_manager.cc:1888] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:06.207919 25308 log.cc:1196] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/c6fa6da27fc745e79963ffa294472d9e
I20240502 01:54:06.208104 25452 raft_consensus.cc:2229] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:06.208199 25097 ts_tablet_manager.cc:1875] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.209280 25308 ts_tablet_manager.cc:1909] T c6fa6da27fc745e79963ffa294472d9e P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:06.210227 25452 raft_consensus.cc:2258] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.210292 25308 tablet_replica.cc:330] T 68e1096b01964772ac02755f1ae21c53 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:06.210444 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet c6fa6da27fc745e79963ffa294472d9e (table rebalance_test_table_14 [id=27c9bf24f886426b92b80b01e4a8d669]) successfully deleted
I20240502 01:54:06.210532 25308 raft_consensus.cc:2229] T 68e1096b01964772ac02755f1ae21c53 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:06.210786 25308 raft_consensus.cc:2258] T 68e1096b01964772ac02755f1ae21c53 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.211498 25308 ts_tablet_manager.cc:1875] T 68e1096b01964772ac02755f1ae21c53 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.217046 25452 ts_tablet_manager.cc:1875] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.217096 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2362b0bec07540f1bc0efedbcf61599c" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:06.217366 25001 raft_consensus.cc:2454] T 2362b0bec07540f1bc0efedbcf61599c P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
W20240502 01:54:06.221020 22298 tablet.cc:2369] T 9f05784e69ff49c8bed9720812a067a3 P c2a28baa7ee2469294d94b7fe5455cff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:54:06.222415 24538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.195:38879, user_credentials={real_user=slave}} blocked reactor thread for 46460.1us
I20240502 01:54:06.222764 25451 raft_consensus.cc:2781] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:06.222936 25451 raft_consensus.cc:482] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:06.224335 25451 raft_consensus.cc:3037] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.227335 25451 raft_consensus.cc:504] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:06.230860 25308 ts_tablet_manager.cc:1888] T 68e1096b01964772ac02755f1ae21c53 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:06.231034 25308 log.cc:1196] T 68e1096b01964772ac02755f1ae21c53 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/68e1096b01964772ac02755f1ae21c53
I20240502 01:54:06.231457 25308 ts_tablet_manager.cc:1909] T 68e1096b01964772ac02755f1ae21c53 P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:06.232789 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 68e1096b01964772ac02755f1ae21c53 (table rebalance_test_table_14 [id=27c9bf24f886426b92b80b01e4a8d669]) successfully deleted
I20240502 01:54:06.236425 25097 ts_tablet_manager.cc:1888] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:06.246233 25100 ts_tablet_manager.cc:1888] T 270e238486f24381815427c4ee2d587a P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:06.246407 25100 log.cc:1196] T 270e238486f24381815427c4ee2d587a P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/270e238486f24381815427c4ee2d587a
I20240502 01:54:06.246863 25100 ts_tablet_manager.cc:1909] T 270e238486f24381815427c4ee2d587a P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:06.247977 25100 tablet_replica.cc:330] T 68e1096b01964772ac02755f1ae21c53 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:06.248222 25100 raft_consensus.cc:2229] T 68e1096b01964772ac02755f1ae21c53 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:06.248201 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 270e238486f24381815427c4ee2d587a (table rebalance_test_table_14 [id=27c9bf24f886426b92b80b01e4a8d669]) successfully deleted
I20240502 01:54:06.248482 25100 raft_consensus.cc:2258] T 68e1096b01964772ac02755f1ae21c53 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.249305 25100 ts_tablet_manager.cc:1875] T 68e1096b01964772ac02755f1ae21c53 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.254670 25452 ts_tablet_manager.cc:1888] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:06.254830 25452 log.cc:1196] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/68e1096b01964772ac02755f1ae21c53
I20240502 01:54:06.255327 25452 ts_tablet_manager.cc:1909] T 68e1096b01964772ac02755f1ae21c53 P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:06.255685 25308 tablet_replica.cc:330] T 270e238486f24381815427c4ee2d587a P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:06.255971 25308 raft_consensus.cc:2229] T 270e238486f24381815427c4ee2d587a P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:06.256304 25308 raft_consensus.cc:2258] T 270e238486f24381815427c4ee2d587a P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.257153 25308 ts_tablet_manager.cc:1875] T 270e238486f24381815427c4ee2d587a P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.258877 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 68e1096b01964772ac02755f1ae21c53 (table rebalance_test_table_14 [id=27c9bf24f886426b92b80b01e4a8d669]) successfully deleted
I20240502 01:54:06.259433 25451 leader_election.cc:288] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:06.260015 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2362b0bec07540f1bc0efedbcf61599c" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:06.260254 24868 raft_consensus.cc:3037] T 2362b0bec07540f1bc0efedbcf61599c P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.260425 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2362b0bec07540f1bc0efedbcf61599c" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:06.260668 25001 raft_consensus.cc:3037] T 2362b0bec07540f1bc0efedbcf61599c P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.262097 25097 log.cc:1196] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/270e238486f24381815427c4ee2d587a
I20240502 01:54:06.263136 24868 raft_consensus.cc:2454] T 2362b0bec07540f1bc0efedbcf61599c P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:06.263478 25001 raft_consensus.cc:2454] T 2362b0bec07540f1bc0efedbcf61599c P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:06.264076 24539 leader_election.cc:302] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:06.265193 25097 ts_tablet_manager.cc:1909] T 270e238486f24381815427c4ee2d587a P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:06.267621 25308 ts_tablet_manager.cc:1888] T 270e238486f24381815427c4ee2d587a P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:06.267777 25308 log.cc:1196] T 270e238486f24381815427c4ee2d587a P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/270e238486f24381815427c4ee2d587a
I20240502 01:54:06.268174 25451 raft_consensus.cc:2781] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:06.268277 25308 ts_tablet_manager.cc:1909] T 270e238486f24381815427c4ee2d587a P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:06.268797 25451 raft_consensus.cc:686] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:54:06.269338 25451 consensus_queue.cc:237] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:06.271912 25100 ts_tablet_manager.cc:1888] T 68e1096b01964772ac02755f1ae21c53 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:06.272096 25100 log.cc:1196] T 68e1096b01964772ac02755f1ae21c53 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/68e1096b01964772ac02755f1ae21c53
I20240502 01:54:06.272552 25100 ts_tablet_manager.cc:1909] T 68e1096b01964772ac02755f1ae21c53 P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:06.273126 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 270e238486f24381815427c4ee2d587a (table rebalance_test_table_14 [id=27c9bf24f886426b92b80b01e4a8d669]) successfully deleted
I20240502 01:54:06.273823 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 68e1096b01964772ac02755f1ae21c53 (table rebalance_test_table_14 [id=27c9bf24f886426b92b80b01e4a8d669]) successfully deleted
I20240502 01:54:06.275043 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 270e238486f24381815427c4ee2d587a (table rebalance_test_table_14 [id=27c9bf24f886426b92b80b01e4a8d669]) successfully deleted
I20240502 01:54:06.279294 21714 catalog_manager.cc:5484] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:06.289026 24868 raft_consensus.cc:1261] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08 [term 2 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 01:54:06.289963 25409 consensus_queue.cc:1035] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:06.299257 25001 raft_consensus.cc:1261] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 01:54:06.300132 25409 consensus_queue.cc:1035] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:06.313046 21714 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:06.323638 21714 catalog_manager.cc:5576] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737) does not have the latest schema for tablet 0689d4851d4143909fe3b7b0d9c7b2d7 (table rebalancer_dynamic_table [id=5ffb0e9de5844f82a2ae0ec1f30c97ef]). Expected version 1 got 0
I20240502 01:54:06.329823 21710 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 0689d4851d4143909fe3b7b0d9c7b2d7 (table rebalancer_dynamic_table [id=5ffb0e9de5844f82a2ae0ec1f30c97ef]). Expected version 1 got 0
I20240502 01:54:06.338971 25459 tablet.cc:1722] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:06.343374 25459 tablet.cc:1620] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77: MemRowSet was empty: no flush needed.
I20240502 01:54:06.353514 25460 tablet.cc:1722] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:06.355569 25460 tablet.cc:1620] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:06.361972 25083 raft_consensus.cc:482] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:06.362202 25083 raft_consensus.cc:504] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:06.363294 25083 leader_election.cc:288] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:06.367643 21714 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=5ffb0e9de5844f82a2ae0ec1f30c97ef] alter complete (version 1)
I20240502 01:54:06.367719 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f05784e69ff49c8bed9720812a067a3" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:06.368005 22249 raft_consensus.cc:2454] T 9f05784e69ff49c8bed9720812a067a3 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:06.368618 21924 leader_election.cc:302] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:06.369297 25083 raft_consensus.cc:2781] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:06.369431 25083 raft_consensus.cc:482] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:06.369526 25083 raft_consensus.cc:3037] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.372649 25083 raft_consensus.cc:504] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:06.376619 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f05784e69ff49c8bed9720812a067a3" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:06.376936 22249 raft_consensus.cc:3037] T 9f05784e69ff49c8bed9720812a067a3 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.376931 25083 leader_election.cc:288] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:06.380820 22249 raft_consensus.cc:2454] T 9f05784e69ff49c8bed9720812a067a3 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:06.383587 21924 leader_election.cc:302] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:06.384663 25408 raft_consensus.cc:2781] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:06.384641 21710 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 0689d4851d4143909fe3b7b0d9c7b2d7 (table rebalancer_dynamic_table [id=5ffb0e9de5844f82a2ae0ec1f30c97ef]). Expected version 1 got 0
I20240502 01:54:06.384821 25408 raft_consensus.cc:686] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:54:06.385196 25408 consensus_queue.cc:237] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:06.386262 25394 raft_consensus.cc:482] T 9f05784e69ff49c8bed9720812a067a3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:06.386428 25394 raft_consensus.cc:504] T 9f05784e69ff49c8bed9720812a067a3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:06.387529 25394 leader_election.cc:288] T 9f05784e69ff49c8bed9720812a067a3 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:06.401911 21714 catalog_manager.cc:5484] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:06.408655 25463 tablet.cc:1722] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:06.409472 25463 tablet.cc:1620] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:06.425082 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f05784e69ff49c8bed9720812a067a3" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:06.425294 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f05784e69ff49c8bed9720812a067a3" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:06.425364 24868 raft_consensus.cc:2454] T 9f05784e69ff49c8bed9720812a067a3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:06.426522 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f05784e69ff49c8bed9720812a067a3" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:54:06.432041 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f05784e69ff49c8bed9720812a067a3" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:06.432415 22249 raft_consensus.cc:2379] T 9f05784e69ff49c8bed9720812a067a3 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7449bb1df214bf4a5f93e4631312f08 in current term 1: Already voted for candidate 06543b05a16f42099bc86db2a0284412 in this term.
I20240502 01:54:06.433178 24803 leader_election.cc:302] T 9f05784e69ff49c8bed9720812a067a3 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7449bb1df214bf4a5f93e4631312f08; no voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff
I20240502 01:54:06.433665 25394 raft_consensus.cc:3037] T 9f05784e69ff49c8bed9720812a067a3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.436717 25394 raft_consensus.cc:2726] T 9f05784e69ff49c8bed9720812a067a3 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:06.482702 21714 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:06.492017 25459 tablet.cc:1722] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:06.492632 25459 tablet.cc:1620] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77: MemRowSet was empty: no flush needed.
I20240502 01:54:06.492805 25463 tablet.cc:1722] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:06.493124 25460 tablet.cc:1722] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:06.493348 25463 tablet.cc:1620] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:06.493570 25460 tablet.cc:1620] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:06.503695 21714 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=5ffb0e9de5844f82a2ae0ec1f30c97ef] alter complete (version 2)
I20240502 01:54:06.529794 21856 consensus_queue.cc:237] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:06.532572 21989 raft_consensus.cc:1261] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:06.532624 22118 raft_consensus.cc:1261] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:06.533139 21791 consensus_peers.cc:485] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 071ad932ce95499fac4d7af7292b42ff. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:06.533381 25410 consensus_queue.cc:1035] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:06.533670 25409 consensus_queue.cc:1035] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:06.533975 25394 consensus_queue.cc:1035] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:06.539086 25394 consensus_queue.cc:1035] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:06.542842 25410 raft_consensus.cc:2932] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:06.543478 21989 raft_consensus.cc:2932] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:06.551787 21710 catalog_manager.cc:5484] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:06.551892 22118 raft_consensus.cc:2932] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:06.606807 21710 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:06.616938 25459 tablet.cc:1722] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:06.617864 25460 tablet.cc:1722] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:06.619302 25463 tablet.cc:1722] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:06.629491 21710 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=5ffb0e9de5844f82a2ae0ec1f30c97ef] alter complete (version 3)
I20240502 01:54:06.682860 25451 consensus_queue.cc:1035] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:06.688463 25307 consensus_queue.cc:1035] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:06.696767 21715 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 01:54:06.697158 21715 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 01:54:06.702576 21715 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet bdb710e59756492d870d1a152a629094
I20240502 01:54:06.703291 21715 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 6f1686dbe93c42b597f32422b6e8ac75
I20240502 01:54:06.703549 21967 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb710e59756492d870d1a152a629094 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:06.703799 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb710e59756492d870d1a152a629094 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:06.703892 25100 tablet_replica.cc:330] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:06.704077 21715 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet b993ed799224421fa6fc29c4cf6dfbd0
I20240502 01:54:06.704130 25100 raft_consensus.cc:2229] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:06.704288 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1686dbe93c42b597f32422b6e8ac75 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:06.704581 25100 raft_consensus.cc:2258] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.704625 25452 tablet_replica.cc:330] T 6f1686dbe93c42b597f32422b6e8ac75 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:06.704903 25452 raft_consensus.cc:2229] T 6f1686dbe93c42b597f32422b6e8ac75 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:06.705199 25452 raft_consensus.cc:2258] T 6f1686dbe93c42b597f32422b6e8ac75 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.705386 25100 ts_tablet_manager.cc:1875] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.706130 25452 ts_tablet_manager.cc:1875] T 6f1686dbe93c42b597f32422b6e8ac75 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.709087 25308 tablet_replica.cc:330] T bdb710e59756492d870d1a152a629094 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:06.709335 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet b993ed799224421fa6fc29c4cf6dfbd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:06.709085 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet b993ed799224421fa6fc29c4cf6dfbd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:06.709432 25308 raft_consensus.cc:2229] T bdb710e59756492d870d1a152a629094 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:06.709764 25308 raft_consensus.cc:2258] T bdb710e59756492d870d1a152a629094 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.710590 25308 ts_tablet_manager.cc:1875] T bdb710e59756492d870d1a152a629094 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.711539 25100 ts_tablet_manager.cc:1888] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:06.711684 25100 log.cc:1196] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/bdb710e59756492d870d1a152a629094
I20240502 01:54:06.712639 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb710e59756492d870d1a152a629094 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:06.713004 25097 tablet_replica.cc:330] T bdb710e59756492d870d1a152a629094 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:06.713244 25097 raft_consensus.cc:2229] T bdb710e59756492d870d1a152a629094 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:06.713374 21967 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1686dbe93c42b597f32422b6e8ac75 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:06.713550 25097 raft_consensus.cc:2258] T bdb710e59756492d870d1a152a629094 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.714082 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1686dbe93c42b597f32422b6e8ac75 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:06.714318 21967 tablet_service.cc:1514] Processing DeleteTablet for tablet b993ed799224421fa6fc29c4cf6dfbd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:06.714421 25097 ts_tablet_manager.cc:1875] T bdb710e59756492d870d1a152a629094 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.714741 25100 ts_tablet_manager.cc:1909] T bdb710e59756492d870d1a152a629094 P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:06.715616 25100 tablet_replica.cc:330] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:06.715822 25100 raft_consensus.cc:2229] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:06.716114 25100 raft_consensus.cc:2258] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.716861 25100 ts_tablet_manager.cc:1875] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.717614 25308 ts_tablet_manager.cc:1888] T bdb710e59756492d870d1a152a629094 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:06.717783 25308 log.cc:1196] T bdb710e59756492d870d1a152a629094 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/bdb710e59756492d870d1a152a629094
I20240502 01:54:06.718245 25308 ts_tablet_manager.cc:1909] T bdb710e59756492d870d1a152a629094 P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:06.719347 25308 tablet_replica.cc:330] T b993ed799224421fa6fc29c4cf6dfbd0 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:06.719550 25308 raft_consensus.cc:2229] T b993ed799224421fa6fc29c4cf6dfbd0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:06.719508 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet bdb710e59756492d870d1a152a629094 (table rebalance_test_table_16 [id=efd69cb962724c4f85c2825debc9eba0]) successfully deleted
I20240502 01:54:06.719820 25308 raft_consensus.cc:2258] T b993ed799224421fa6fc29c4cf6dfbd0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.720554 25308 ts_tablet_manager.cc:1875] T b993ed799224421fa6fc29c4cf6dfbd0 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.721589 25100 ts_tablet_manager.cc:1888] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:06.721735 25100 log.cc:1196] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/6f1686dbe93c42b597f32422b6e8ac75
I20240502 01:54:06.722080 25100 ts_tablet_manager.cc:1909] T 6f1686dbe93c42b597f32422b6e8ac75 P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:06.723340 25097 ts_tablet_manager.cc:1888] T bdb710e59756492d870d1a152a629094 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:06.723493 25097 log.cc:1196] T bdb710e59756492d870d1a152a629094 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/bdb710e59756492d870d1a152a629094
I20240502 01:54:06.723520 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet bdb710e59756492d870d1a152a629094 (table rebalance_test_table_16 [id=efd69cb962724c4f85c2825debc9eba0]) successfully deleted
I20240502 01:54:06.723788 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 6f1686dbe93c42b597f32422b6e8ac75 (table rebalance_test_table_16 [id=efd69cb962724c4f85c2825debc9eba0]) successfully deleted
I20240502 01:54:06.723933 25097 ts_tablet_manager.cc:1909] T bdb710e59756492d870d1a152a629094 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:06.723961 25100 tablet_replica.cc:330] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:06.724161 25100 raft_consensus.cc:2229] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:06.724501 25100 raft_consensus.cc:2258] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.725015 25097 tablet_replica.cc:330] T 6f1686dbe93c42b597f32422b6e8ac75 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:06.725127 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet bdb710e59756492d870d1a152a629094 (table rebalance_test_table_16 [id=efd69cb962724c4f85c2825debc9eba0]) successfully deleted
I20240502 01:54:06.725215 25097 raft_consensus.cc:2229] T 6f1686dbe93c42b597f32422b6e8ac75 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:06.725291 25100 ts_tablet_manager.cc:1875] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.725462 25097 raft_consensus.cc:2258] T 6f1686dbe93c42b597f32422b6e8ac75 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.726197 25097 ts_tablet_manager.cc:1875] T 6f1686dbe93c42b597f32422b6e8ac75 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.726789 25308 ts_tablet_manager.cc:1888] T b993ed799224421fa6fc29c4cf6dfbd0 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:06.726931 25308 log.cc:1196] T b993ed799224421fa6fc29c4cf6dfbd0 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/b993ed799224421fa6fc29c4cf6dfbd0
I20240502 01:54:06.727420 25308 ts_tablet_manager.cc:1909] T b993ed799224421fa6fc29c4cf6dfbd0 P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:06.728523 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet b993ed799224421fa6fc29c4cf6dfbd0 (table rebalance_test_table_16 [id=efd69cb962724c4f85c2825debc9eba0]) successfully deleted
I20240502 01:54:06.729533 25452 ts_tablet_manager.cc:1888] T 6f1686dbe93c42b597f32422b6e8ac75 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:06.730648 25452 log.cc:1196] T 6f1686dbe93c42b597f32422b6e8ac75 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/6f1686dbe93c42b597f32422b6e8ac75
I20240502 01:54:06.731174 25452 ts_tablet_manager.cc:1909] T 6f1686dbe93c42b597f32422b6e8ac75 P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:06.731442 25100 ts_tablet_manager.cc:1888] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:06.732388 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 6f1686dbe93c42b597f32422b6e8ac75 (table rebalance_test_table_16 [id=efd69cb962724c4f85c2825debc9eba0]) successfully deleted
I20240502 01:54:06.732805 25452 tablet_replica.cc:330] T b993ed799224421fa6fc29c4cf6dfbd0 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:06.733027 25100 log.cc:1196] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/b993ed799224421fa6fc29c4cf6dfbd0
I20240502 01:54:06.733065 25452 raft_consensus.cc:2229] T b993ed799224421fa6fc29c4cf6dfbd0 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:06.733353 25452 raft_consensus.cc:2258] T b993ed799224421fa6fc29c4cf6dfbd0 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.734192 25452 ts_tablet_manager.cc:1875] T b993ed799224421fa6fc29c4cf6dfbd0 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.734498 25100 ts_tablet_manager.cc:1909] T b993ed799224421fa6fc29c4cf6dfbd0 P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:06.738915 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet b993ed799224421fa6fc29c4cf6dfbd0 (table rebalance_test_table_16 [id=efd69cb962724c4f85c2825debc9eba0]) successfully deleted
I20240502 01:54:06.739217 21714 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:06.739445 21714 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:06.743809 21714 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 0689d4851d4143909fe3b7b0d9c7b2d7
I20240502 01:54:06.744724 24848 tablet_service.cc:1514] Processing DeleteTablet for tablet 0689d4851d4143909fe3b7b0d9c7b2d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:51592
I20240502 01:54:06.746137 24980 tablet_service.cc:1514] Processing DeleteTablet for tablet 0689d4851d4143909fe3b7b0d9c7b2d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:39964
I20240502 01:54:06.746806 25528 tablet_replica.cc:330] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:54:06.747020 25528 raft_consensus.cc:2229] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:06.747277 25528 raft_consensus.cc:2258] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.748136 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 0689d4851d4143909fe3b7b0d9c7b2d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:06 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:06.748353 25528 ts_tablet_manager.cc:1875] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.752295 25452 ts_tablet_manager.cc:1888] T b993ed799224421fa6fc29c4cf6dfbd0 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:06.752458 25452 log.cc:1196] T b993ed799224421fa6fc29c4cf6dfbd0 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/b993ed799224421fa6fc29c4cf6dfbd0
I20240502 01:54:06.753005 25452 ts_tablet_manager.cc:1909] T b993ed799224421fa6fc29c4cf6dfbd0 P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:06.754319 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet b993ed799224421fa6fc29c4cf6dfbd0 (table rebalance_test_table_16 [id=efd69cb962724c4f85c2825debc9eba0]) successfully deleted
I20240502 01:54:06.760632 25528 ts_tablet_manager.cc:1888] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.4
I20240502 01:54:06.761358 25097 ts_tablet_manager.cc:1888] T 6f1686dbe93c42b597f32422b6e8ac75 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:06.761626 25097 log.cc:1196] T 6f1686dbe93c42b597f32422b6e8ac75 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/6f1686dbe93c42b597f32422b6e8ac75
I20240502 01:54:06.762243 25097 ts_tablet_manager.cc:1909] T 6f1686dbe93c42b597f32422b6e8ac75 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:06.763841 25527 tablet_replica.cc:330] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08: stopping tablet replica
I20240502 01:54:06.764389 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 6f1686dbe93c42b597f32422b6e8ac75 (table rebalance_test_table_16 [id=efd69cb962724c4f85c2825debc9eba0]) successfully deleted
I20240502 01:54:06.765010 25097 tablet_replica.cc:330] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:06.765350 25527 raft_consensus.cc:2229] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:06.765708 25097 raft_consensus.cc:2229] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 2 LEADER]: Raft consensus shutting down.
I20240502 01:54:06.766062 25527 raft_consensus.cc:2258] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.766613 25097 raft_consensus.cc:2258] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:06.769543 25527 ts_tablet_manager.cc:1875] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.770207 25097 ts_tablet_manager.cc:1875] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:06.779675 21967 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:06.779924 25528 log.cc:1196] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/0689d4851d4143909fe3b7b0d9c7b2d7
I20240502 01:54:06.780553 25528 ts_tablet_manager.cc:1909] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting consensus metadata
I20240502 01:54:06.785348 25097 ts_tablet_manager.cc:1888] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.4
I20240502 01:54:06.789916 25527 ts_tablet_manager.cc:1888] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.4
I20240502 01:54:06.790252 25097 log.cc:1196] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/0689d4851d4143909fe3b7b0d9c7b2d7
I20240502 01:54:06.790462 25527 log.cc:1196] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/wals/0689d4851d4143909fe3b7b0d9c7b2d7
I20240502 01:54:06.791050 25527 ts_tablet_manager.cc:1909] T 0689d4851d4143909fe3b7b0d9c7b2d7 P d7449bb1df214bf4a5f93e4631312f08: Deleting consensus metadata
I20240502 01:54:06.791615 25097 ts_tablet_manager.cc:1909] T 0689d4851d4143909fe3b7b0d9c7b2d7 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:06.792955 21700 catalog_manager.cc:4830] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): tablet 0689d4851d4143909fe3b7b0d9c7b2d7 (table rebalancer_dynamic_table_new_name [id=5ffb0e9de5844f82a2ae0ec1f30c97ef]) successfully deleted
I20240502 01:54:06.794349 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 0689d4851d4143909fe3b7b0d9c7b2d7 (table rebalancer_dynamic_table_new_name [id=5ffb0e9de5844f82a2ae0ec1f30c97ef]) successfully deleted
I20240502 01:54:06.814977 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet 0689d4851d4143909fe3b7b0d9c7b2d7 (table rebalancer_dynamic_table_new_name [id=5ffb0e9de5844f82a2ae0ec1f30c97ef]) successfully deleted
I20240502 01:54:06.830544 24581 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:06.834117 24714 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:06.837731 22231 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:06.843313 21836 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:06.849561 21715 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:06.864437 24980 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:54:06.866102 24714 tablet_service.cc:1467] Processing CreateTablet for tablet a9ae84af0ef041a792f072c96644b36a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=97f910a299c84acbb8c29efdc3a2a411]), partition=
I20240502 01:54:06.866181 24980 tablet_service.cc:1467] Processing CreateTablet for tablet a9ae84af0ef041a792f072c96644b36a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=97f910a299c84acbb8c29efdc3a2a411]), partition=
I20240502 01:54:06.866534 24714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ae84af0ef041a792f072c96644b36a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.866607 24980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ae84af0ef041a792f072c96644b36a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.873755 22231 tablet_service.cc:1467] Processing CreateTablet for tablet a9ae84af0ef041a792f072c96644b36a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=97f910a299c84acbb8c29efdc3a2a411]), partition=
I20240502 01:54:06.874189 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ae84af0ef041a792f072c96644b36a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.909942 21709 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:06.918028 25413 consensus_queue.cc:1035] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:06.919497 22100 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:06.932359 24981 tablet_service.cc:1467] Processing CreateTablet for tablet 96c5ae723e7a43a0b91a743b0c192a67 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=de52a7578d964d1ba0e315c9f1e99b21]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:06.932922 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c5ae723e7a43a0b91a743b0c192a67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.933502 21967 tablet_service.cc:1467] Processing CreateTablet for tablet 96c5ae723e7a43a0b91a743b0c192a67 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=de52a7578d964d1ba0e315c9f1e99b21]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:06.933964 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c5ae723e7a43a0b91a743b0c192a67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.934240 24581 tablet_service.cc:1467] Processing CreateTablet for tablet 564f981f7f7c4f2480c4fd1e5b5c1ad8 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=de52a7578d964d1ba0e315c9f1e99b21]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:06.934372 24979 tablet_service.cc:1467] Processing CreateTablet for tablet 564f981f7f7c4f2480c4fd1e5b5c1ad8 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=de52a7578d964d1ba0e315c9f1e99b21]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:06.934731 24581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564f981f7f7c4f2480c4fd1e5b5c1ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.934890 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564f981f7f7c4f2480c4fd1e5b5c1ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.937712 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 96c5ae723e7a43a0b91a743b0c192a67 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=de52a7578d964d1ba0e315c9f1e99b21]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:06.938185 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c5ae723e7a43a0b91a743b0c192a67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.943181 24848 tablet_service.cc:1467] Processing CreateTablet for tablet 564f981f7f7c4f2480c4fd1e5b5c1ad8 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=de52a7578d964d1ba0e315c9f1e99b21]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:06.943825 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564f981f7f7c4f2480c4fd1e5b5c1ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.949090 25550 tablet_bootstrap.cc:492] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:06.951193 24582 tablet_service.cc:1467] Processing CreateTablet for tablet e00feee2662a48d8a1a57f9e52275154 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=de52a7578d964d1ba0e315c9f1e99b21]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:06.951437 24981 tablet_service.cc:1467] Processing CreateTablet for tablet e00feee2662a48d8a1a57f9e52275154 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=de52a7578d964d1ba0e315c9f1e99b21]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:06.951653 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00feee2662a48d8a1a57f9e52275154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.952281 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00feee2662a48d8a1a57f9e52275154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.962642 25553 tablet_bootstrap.cc:492] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:06.966847 25550 tablet_bootstrap.cc:654] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:06.968489 25553 tablet_bootstrap.cc:654] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:06.975620 24847 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:06.983280 25550 tablet_bootstrap.cc:492] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:06.984762 25550 ts_tablet_manager.cc:1367] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.003s
I20240502 01:54:06.986271 25550 raft_consensus.cc:348] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:06.990109 25553 tablet_bootstrap.cc:492] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:06.990309 25550 raft_consensus.cc:374] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:06.990511 25550 raft_consensus.cc:729] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:06.991067 25553 ts_tablet_manager.cc:1367] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.000s
I20240502 01:54:06.991451 25550 consensus_queue.cc:260] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:06.992621 25553 raft_consensus.cc:348] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:06.993196 25544 tablet_bootstrap.cc:492] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:06.994496 25550 ts_tablet_manager.cc:1398] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:54:06.994835 25553 raft_consensus.cc:374] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:06.995007 25553 raft_consensus.cc:729] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:06.995362 25553 consensus_queue.cc:260] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:06.996089 25553 ts_tablet_manager.cc:1398] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:06.996650 25553 tablet_bootstrap.cc:492] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:06.999752 25307 raft_consensus.cc:482] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:06.999946 25307 raft_consensus.cc:504] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:07.000730 25307 leader_election.cc:288] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:07.000941 21836 tablet_service.cc:1467] Processing CreateTablet for tablet e00feee2662a48d8a1a57f9e52275154 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=de52a7578d964d1ba0e315c9f1e99b21]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:07.001416 25553 tablet_bootstrap.cc:654] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.001760 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564f981f7f7c4f2480c4fd1e5b5c1ad8" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
W20240502 01:54:07.002894 24539 leader_election.cc:341] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:07.004098 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00feee2662a48d8a1a57f9e52275154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:07.005784 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564f981f7f7c4f2480c4fd1e5b5c1ad8" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
W20240502 01:54:07.007175 24537 leader_election.cc:341] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:07.007356 24537 leader_election.cc:302] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 41c70f373c9d49a49891b03a12c6cb2b, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:07.008142 25464 consensus_queue.cc:1035] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:07.008683 25307 raft_consensus.cc:2726] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:07.010416 25553 tablet_bootstrap.cc:492] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:07.011168 25553 ts_tablet_manager.cc:1367] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.004s
I20240502 01:54:07.012261 25553 raft_consensus.cc:348] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:07.014317 25553 raft_consensus.cc:374] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.014616 25553 raft_consensus.cc:729] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.015363 25553 consensus_queue.cc:260] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:07.016513 25553 ts_tablet_manager.cc:1398] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:54:07.063160 25554 tablet_bootstrap.cc:492] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:07.066517 25562 tablet_bootstrap.cc:492] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:07.067850 25559 ts_tablet_manager.cc:902] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08: Initiating tablet copy from peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
W20240502 01:54:07.070086 24933 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.193:33785, user_credentials={real_user=slave}} blocked reactor thread for 58806.7us
I20240502 01:54:07.076184 25546 tablet_bootstrap.cc:492] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:07.077479 25544 tablet_bootstrap.cc:654] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.078626 25546 tablet_bootstrap.cc:654] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.085613 25545 tablet_bootstrap.cc:492] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:07.103493 25554 tablet_bootstrap.cc:654] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.104038 25544 tablet_bootstrap.cc:492] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:07.104310 25544 ts_tablet_manager.cc:1367] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.111s user 0.005s sys 0.000s
I20240502 01:54:07.105558 25562 tablet_bootstrap.cc:654] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.105683 25545 tablet_bootstrap.cc:654] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.105731 25544 raft_consensus.cc:348] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:07.106180 25544 raft_consensus.cc:374] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.106336 25544 raft_consensus.cc:729] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.106922 25544 consensus_queue.cc:260] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:07.109011 25544 ts_tablet_manager.cc:1398] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:54:07.109699 25544 tablet_bootstrap.cc:492] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:07.119592 25544 tablet_bootstrap.cc:654] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.120699 25546 tablet_bootstrap.cc:492] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:07.120997 25546 ts_tablet_manager.cc:1367] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.000s
I20240502 01:54:07.122056 25546 raft_consensus.cc:348] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:07.122579 25546 raft_consensus.cc:374] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.122740 25546 raft_consensus.cc:729] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.123083 25546 consensus_queue.cc:260] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:07.125046 25546 ts_tablet_manager.cc:1398] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:07.125584 25546 tablet_bootstrap.cc:492] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:07.127429 25562 tablet_bootstrap.cc:492] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:07.127627 25562 ts_tablet_manager.cc:1367] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.000s
I20240502 01:54:07.128535 25562 raft_consensus.cc:348] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:07.128773 25562 raft_consensus.cc:374] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.128868 25562 raft_consensus.cc:729] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.129472 25562 consensus_queue.cc:260] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:07.130290 25410 raft_consensus.cc:482] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:07.130431 25410 raft_consensus.cc:504] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:07.131147 25410 leader_election.cc:288] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:07.132012 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00feee2662a48d8a1a57f9e52275154" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
W20240502 01:54:07.132674 21793 leader_election.cc:341] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:07.133029 25562 ts_tablet_manager.cc:1398] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240502 01:54:07.135111 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00feee2662a48d8a1a57f9e52275154" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:07.135383 24603 raft_consensus.cc:2454] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:54:07.135840 21791 leader_election.cc:302] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 759257ea7bef415ca698e60d82199d77; no voters: 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:07.136287 25410 raft_consensus.cc:2781] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:07.136406 25410 raft_consensus.cc:482] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:07.136502 25410 raft_consensus.cc:3037] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.139331 25410 raft_consensus.cc:504] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:07.140584 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00feee2662a48d8a1a57f9e52275154" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
W20240502 01:54:07.141687 21793 leader_election.cc:341] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:07.142146 25410 leader_election.cc:288] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:07.145396 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00feee2662a48d8a1a57f9e52275154" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:07.145646 24603 raft_consensus.cc:3037] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.149006 24603 raft_consensus.cc:2454] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:54:07.149570 21791 leader_election.cc:302] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 759257ea7bef415ca698e60d82199d77; no voters: 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:07.150734 25546 tablet_bootstrap.cc:654] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.153412 25409 raft_consensus.cc:2781] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:07.153607 25409 raft_consensus.cc:686] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:54:07.153949 25409 consensus_queue.cc:237] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:07.163345 25174 raft_consensus.cc:482] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:07.163604 25174 raft_consensus.cc:504] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:07.169401 25413 raft_consensus.cc:482] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:07.169611 25413 raft_consensus.cc:504] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:07.170569 25545 tablet_bootstrap.cc:492] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:07.170588 25413 leader_election.cc:288] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:07.170750 25545 ts_tablet_manager.cc:1367] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.086s user 0.004s sys 0.000s
I20240502 01:54:07.171653 25545 raft_consensus.cc:348] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:07.171880 25545 raft_consensus.cc:374] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.171973 25545 raft_consensus.cc:729] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.171972 25559 tablet_copy_client.cc:308] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.193:33785
I20240502 01:54:07.172235 25545 consensus_queue.cc:260] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:07.172978 25554 tablet_bootstrap.cc:492] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:07.172856 25545 ts_tablet_manager.cc:1398] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:07.173151 25554 ts_tablet_manager.cc:1367] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.110s user 0.005s sys 0.000s
I20240502 01:54:07.175841 25554 raft_consensus.cc:348] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:07.176066 25554 raft_consensus.cc:374] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.176167 25554 raft_consensus.cc:729] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.176151 25174 leader_election.cc:288] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:07.176169 21878 tablet_copy_service.cc:140] P 759257ea7bef415ca698e60d82199d77: Received BeginTabletCopySession request for tablet 071ad932ce95499fac4d7af7292b42ff from peer d7449bb1df214bf4a5f93e4631312f08 ({username='slave'} at 127.14.212.199:41477)
I20240502 01:54:07.176396 21878 tablet_copy_service.cc:161] P 759257ea7bef415ca698e60d82199d77: Beginning new tablet copy session on tablet 071ad932ce95499fac4d7af7292b42ff from peer d7449bb1df214bf4a5f93e4631312f08 at {username='slave'} at 127.14.212.199:41477: session id = d7449bb1df214bf4a5f93e4631312f08-071ad932ce95499fac4d7af7292b42ff
I20240502 01:54:07.179674 25554 consensus_queue.cc:260] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:07.180763 25322 raft_consensus.cc:482] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:07.180960 25322 raft_consensus.cc:504] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:07.181816 25322 leader_election.cc:288] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:07.181807 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ae84af0ef041a792f072c96644b36a" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:07.182077 25001 raft_consensus.cc:2454] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:07.183564 24671 leader_election.cc:302] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:07.184082 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564f981f7f7c4f2480c4fd1e5b5c1ad8" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
W20240502 01:54:07.184767 24804 leader_election.cc:341] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:07.185130 25174 raft_consensus.cc:2781] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:07.185246 25174 raft_consensus.cc:482] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:07.185372 25174 raft_consensus.cc:3037] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.186926 21878 tablet_copy_source_session.cc:205] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:07.189543 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564f981f7f7c4f2480c4fd1e5b5c1ad8" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:07.189854 24603 raft_consensus.cc:2454] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:07.190330 24802 leader_election.cc:302] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08; no voters: 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:07.190979 25322 raft_consensus.cc:2781] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:07.191078 25322 raft_consensus.cc:482] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:07.191181 25322 raft_consensus.cc:3037] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.194269 25322 raft_consensus.cc:504] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:07.196383 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564f981f7f7c4f2480c4fd1e5b5c1ad8" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
W20240502 01:54:07.197079 24804 leader_election.cc:341] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:07.197314 25322 leader_election.cc:288] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:07.197660 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564f981f7f7c4f2480c4fd1e5b5c1ad8" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:07.197903 24603 raft_consensus.cc:3037] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.198401 25559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071ad932ce95499fac4d7af7292b42ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:07.200778 24603 raft_consensus.cc:2454] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:07.201140 25174 raft_consensus.cc:504] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:07.202963 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ae84af0ef041a792f072c96644b36a" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:07.203265 25001 raft_consensus.cc:3037] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.206754 25001 raft_consensus.cc:2454] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:07.207382 24671 leader_election.cc:302] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:07.210424 25174 leader_election.cc:288] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:07.211026 25174 raft_consensus.cc:2781] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:07.211638 25174 raft_consensus.cc:686] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:07.212350 25174 consensus_queue.cc:237] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:07.213798 24802 leader_election.cc:302] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08; no voters: 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:07.214444 25322 raft_consensus.cc:2781] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:07.223160 25554 ts_tablet_manager.cc:1398] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.050s user 0.002s sys 0.000s
I20240502 01:54:07.224664 21709 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 01:54:07.224972 21709 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 01:54:07.225596 25544 tablet_bootstrap.cc:492] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:07.229887 21709 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 4 replicas of tablet 071ad932ce95499fac4d7af7292b42ff
I20240502 01:54:07.232774 25544 ts_tablet_manager.cc:1367] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.123s user 0.001s sys 0.019s
I20240502 01:54:07.232825 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 071ad932ce95499fac4d7af7292b42ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:07 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:07.233165 25452 tablet_replica.cc:330] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:07.233436 25452 raft_consensus.cc:2229] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:07.233562 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 071ad932ce95499fac4d7af7292b42ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:07 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:07.233762 25452 raft_consensus.cc:2258] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.233911 25097 tablet_replica.cc:330] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:07.234117 25097 raft_consensus.cc:2229] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:07.234630 25097 raft_consensus.cc:2258] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.236188 25452 ts_tablet_manager.cc:1875] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:07.237880 25544 raft_consensus.cc:348] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:07.238157 25544 raft_consensus.cc:374] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.240118 25544 raft_consensus.cc:729] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.240398 25544 consensus_queue.cc:260] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:07.242226 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ae84af0ef041a792f072c96644b36a" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:07.242591 22250 raft_consensus.cc:2454] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:07.242226 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ae84af0ef041a792f072c96644b36a" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:07.243517 22249 raft_consensus.cc:3037] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.244482 25544 ts_tablet_manager.cc:1398] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 01:54:07.246497 22249 raft_consensus.cc:2454] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:07.248193 25544 tablet_bootstrap.cc:492] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:07.250025 25544 tablet_bootstrap.cc:654] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.259858 25097 ts_tablet_manager.cc:1875] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:07.265882 25322 raft_consensus.cc:686] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:07.266235 25322 consensus_queue.cc:237] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:07.268520 21710 catalog_manager.cc:5484] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:07.269825 21715 catalog_manager.cc:5484] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:07.275691 24848 tablet_service.cc:1514] Processing DeleteTablet for tablet 071ad932ce95499fac4d7af7292b42ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:07 UTC) from {username='slave'} at 127.0.0.1:51592
I20240502 01:54:07.276005 21709 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 5eb367ab8254423c93c7f33f885d374f
W20240502 01:54:07.276752 21700 catalog_manager.cc:4797] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): delete failed for tablet 071ad932ce95499fac4d7af7292b42ff because the tablet was not found. No further retry: Not found: Tablet not found: 071ad932ce95499fac4d7af7292b42ff
I20240502 01:54:07.277084 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 5eb367ab8254423c93c7f33f885d374f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:07 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:07.277803 25559 tablet_copy_client.cc:783] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:07.278334 25559 tablet_copy_client.cc:647] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:07.283401 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet 5eb367ab8254423c93c7f33f885d374f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:07 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:07.284163 21709 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 37e185badcf64f078e0df67837cfe054
I20240502 01:54:07.285624 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e185badcf64f078e0df67837cfe054 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:07 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:07.298470 25452 ts_tablet_manager.cc:1888] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:07.299613 25452 log.cc:1196] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/071ad932ce95499fac4d7af7292b42ff
I20240502 01:54:07.300360 25452 ts_tablet_manager.cc:1909] T 071ad932ce95499fac4d7af7292b42ff P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:07.303982 25587 tablet_replica.cc:330] T 5eb367ab8254423c93c7f33f885d374f P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:07.305294 25452 tablet_replica.cc:330] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:07.305766 25587 raft_consensus.cc:2229] T 5eb367ab8254423c93c7f33f885d374f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:07.306365 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 071ad932ce95499fac4d7af7292b42ff (table rebalance_test_table_18 [id=596028c83e784a669c0b5819444e54be]) successfully deleted
I20240502 01:54:07.306893 25587 raft_consensus.cc:2258] T 5eb367ab8254423c93c7f33f885d374f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.306479 25452 raft_consensus.cc:2229] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:07.307140 21710 catalog_manager.cc:5484] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:07.307914 25452 raft_consensus.cc:2258] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.311501 25587 ts_tablet_manager.cc:1875] T 5eb367ab8254423c93c7f33f885d374f P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:07.313781 25546 tablet_bootstrap.cc:492] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:07.314215 25546 ts_tablet_manager.cc:1367] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.189s user 0.011s sys 0.011s
I20240502 01:54:07.315577 25546 raft_consensus.cc:348] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:07.316082 25546 raft_consensus.cc:374] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.316249 25546 raft_consensus.cc:729] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.316648 25546 consensus_queue.cc:260] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:07.317865 25546 ts_tablet_manager.cc:1398] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:07.320655 25452 ts_tablet_manager.cc:1875] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:07.339084 25581 raft_consensus.cc:482] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:07.339274 25581 raft_consensus.cc:504] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:07.340186 25581 leader_election.cc:288] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:07.341076 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c5ae723e7a43a0b91a743b0c192a67" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:07.341318 25001 raft_consensus.cc:2454] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:07.342113 25452 ts_tablet_manager.cc:1888] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:07.342278 25452 log.cc:1196] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/5eb367ab8254423c93c7f33f885d374f
I20240502 01:54:07.342222 21987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c5ae723e7a43a0b91a743b0c192a67" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:54:07.342481 21987 raft_consensus.cc:2454] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:07.342772 25452 ts_tablet_manager.cc:1909] T 5eb367ab8254423c93c7f33f885d374f P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:07.343163 24671 leader_election.cc:302] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:07.343910 25581 raft_consensus.cc:2781] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:07.344022 25581 raft_consensus.cc:482] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:07.344096 25581 raft_consensus.cc:3037] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.346254 21967 tablet_service.cc:1514] Processing DeleteTablet for tablet 071ad932ce95499fac4d7af7292b42ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:07 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:07.347026 25581 raft_consensus.cc:504] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:07.348389 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c5ae723e7a43a0b91a743b0c192a67" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:07.348599 25001 raft_consensus.cc:3037] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.351541 25001 raft_consensus.cc:2454] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:07.352157 24671 leader_election.cc:302] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:07.352504 25174 raft_consensus.cc:2781] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:07.352857 21969 tablet_service.cc:1514] Processing DeleteTablet for tablet 5eb367ab8254423c93c7f33f885d374f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:07 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:07.353649 25587 ts_tablet_manager.cc:1888] T 5eb367ab8254423c93c7f33f885d374f P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:07.353688 21969 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e185badcf64f078e0df67837cfe054 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:07 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:07.354413 25587 log.cc:1196] T 5eb367ab8254423c93c7f33f885d374f P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/5eb367ab8254423c93c7f33f885d374f
I20240502 01:54:07.354475 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c5ae723e7a43a0b91a743b0c192a67" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:07.354878 25001 raft_consensus.cc:2379] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 06543b05a16f42099bc86db2a0284412 in current term 1: Already voted for candidate 28157bc3e678428fb5f93c077d528513 in this term.
I20240502 01:54:07.354984 25587 ts_tablet_manager.cc:1909] T 5eb367ab8254423c93c7f33f885d374f P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:07.356328 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 5eb367ab8254423c93c7f33f885d374f (table rebalance_test_table_18 [id=596028c83e784a669c0b5819444e54be]) successfully deleted
I20240502 01:54:07.356421 21987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c5ae723e7a43a0b91a743b0c192a67" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:54:07.356642 21987 raft_consensus.cc:3037] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.356673 25452 tablet_replica.cc:330] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:07.356806 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 5eb367ab8254423c93c7f33f885d374f (table rebalance_test_table_18 [id=596028c83e784a669c0b5819444e54be]) successfully deleted
I20240502 01:54:07.356941 25452 raft_consensus.cc:2229] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:07.357379 25452 raft_consensus.cc:2258] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.358093 25581 leader_election.cc:288] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:07.358325 25452 ts_tablet_manager.cc:1875] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:07.359933 21987 raft_consensus.cc:2454] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:07.363132 25588 tablet_replica.cc:330] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:07.368899 25452 ts_tablet_manager.cc:1888] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:07.369810 25174 raft_consensus.cc:686] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:07.370242 25174 consensus_queue.cc:237] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:07.374629 25588 raft_consensus.cc:2229] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:07.375089 25588 raft_consensus.cc:2258] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.375082 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c5ae723e7a43a0b91a743b0c192a67" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:07.375872 21923 leader_election.cc:302] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06543b05a16f42099bc86db2a0284412; no voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:07.376268 25588 ts_tablet_manager.cc:1875] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:07.376385 25413 raft_consensus.cc:2726] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:07.381072 25452 log.cc:1196] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/37e185badcf64f078e0df67837cfe054
I20240502 01:54:07.381718 25452 ts_tablet_manager.cc:1909] T 37e185badcf64f078e0df67837cfe054 P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:07.383419 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 37e185badcf64f078e0df67837cfe054 (table rebalance_test_table_18 [id=596028c83e784a669c0b5819444e54be]) successfully deleted
I20240502 01:54:07.386161 25544 tablet_bootstrap.cc:492] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:07.386335 25544 ts_tablet_manager.cc:1367] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.138s user 0.040s sys 0.031s
I20240502 01:54:07.387141 25544 raft_consensus.cc:348] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:07.387349 25544 raft_consensus.cc:374] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.387431 25544 raft_consensus.cc:729] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.387673 25544 consensus_queue.cc:260] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
W20240502 01:54:07.387794 21791 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.14.212.199:35737, user_credentials={real_user=slave}} blocked reactor thread for 101000us
I20240502 01:54:07.389775 25544 ts_tablet_manager.cc:1398] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:54:07.390092 25544 tablet_bootstrap.cc:492] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:07.392426 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e185badcf64f078e0df67837cfe054 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:07 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:07.392781 25544 tablet_bootstrap.cc:654] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.396692 25097 ts_tablet_manager.cc:1888] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:07.397308 25588 ts_tablet_manager.cc:1888] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:07.397477 25588 log.cc:1196] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/071ad932ce95499fac4d7af7292b42ff
I20240502 01:54:07.398042 25588 ts_tablet_manager.cc:1909] T 071ad932ce95499fac4d7af7292b42ff P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:07.399147 25588 tablet_replica.cc:330] T 5eb367ab8254423c93c7f33f885d374f P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:07.399391 25588 raft_consensus.cc:2229] T 5eb367ab8254423c93c7f33f885d374f P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:07.399650 25588 raft_consensus.cc:2258] T 5eb367ab8254423c93c7f33f885d374f P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.400435 25588 ts_tablet_manager.cc:1875] T 5eb367ab8254423c93c7f33f885d374f P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:07.400748 25097 log.cc:1196] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/071ad932ce95499fac4d7af7292b42ff
I20240502 01:54:07.406781 21710 catalog_manager.cc:5484] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:07.408301 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 071ad932ce95499fac4d7af7292b42ff (table rebalance_test_table_18 [id=596028c83e784a669c0b5819444e54be]) successfully deleted
I20240502 01:54:07.408851 25544 tablet_bootstrap.cc:492] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:07.409073 25544 ts_tablet_manager.cc:1367] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.004s
I20240502 01:54:07.409327 25097 ts_tablet_manager.cc:1909] T 071ad932ce95499fac4d7af7292b42ff P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:07.409894 25544 raft_consensus.cc:348] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:07.410135 25544 raft_consensus.cc:374] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.410131 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 071ad932ce95499fac4d7af7292b42ff (table rebalance_test_table_18 [id=596028c83e784a669c0b5819444e54be]) successfully deleted
I20240502 01:54:07.410212 25544 raft_consensus.cc:729] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.410481 25544 consensus_queue.cc:260] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:07.411067 25544 ts_tablet_manager.cc:1398] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:54:07.412470 25097 tablet_replica.cc:330] T 37e185badcf64f078e0df67837cfe054 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:07.412721 25097 raft_consensus.cc:2229] T 37e185badcf64f078e0df67837cfe054 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:07.413033 25097 raft_consensus.cc:2258] T 37e185badcf64f078e0df67837cfe054 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.413777 25097 ts_tablet_manager.cc:1875] T 37e185badcf64f078e0df67837cfe054 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:07.414171 25588 ts_tablet_manager.cc:1888] T 5eb367ab8254423c93c7f33f885d374f P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:07.414340 25588 log.cc:1196] T 5eb367ab8254423c93c7f33f885d374f P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/5eb367ab8254423c93c7f33f885d374f
I20240502 01:54:07.414825 25588 ts_tablet_manager.cc:1909] T 5eb367ab8254423c93c7f33f885d374f P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:07.415846 25588 tablet_replica.cc:330] T 37e185badcf64f078e0df67837cfe054 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:07.415941 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 5eb367ab8254423c93c7f33f885d374f (table rebalance_test_table_18 [id=596028c83e784a669c0b5819444e54be]) successfully deleted
I20240502 01:54:07.416078 25588 raft_consensus.cc:2229] T 37e185badcf64f078e0df67837cfe054 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:07.416353 25588 raft_consensus.cc:2258] T 37e185badcf64f078e0df67837cfe054 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.417083 25588 ts_tablet_manager.cc:1875] T 37e185badcf64f078e0df67837cfe054 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:07.420341 25097 ts_tablet_manager.cc:1888] T 37e185badcf64f078e0df67837cfe054 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:07.420563 25097 log.cc:1196] T 37e185badcf64f078e0df67837cfe054 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/37e185badcf64f078e0df67837cfe054
I20240502 01:54:07.421134 25097 ts_tablet_manager.cc:1909] T 37e185badcf64f078e0df67837cfe054 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:07.426414 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 37e185badcf64f078e0df67837cfe054 (table rebalance_test_table_18 [id=596028c83e784a669c0b5819444e54be]) successfully deleted
I20240502 01:54:07.428503 25588 ts_tablet_manager.cc:1888] T 37e185badcf64f078e0df67837cfe054 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:54:07.428721 25588 log.cc:1196] T 37e185badcf64f078e0df67837cfe054 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/37e185badcf64f078e0df67837cfe054
I20240502 01:54:07.429198 25588 ts_tablet_manager.cc:1909] T 37e185badcf64f078e0df67837cfe054 P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:07.429561 25559 tablet_copy_client.cc:523] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:07.433887 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 37e185badcf64f078e0df67837cfe054 (table rebalance_test_table_18 [id=596028c83e784a669c0b5819444e54be]) successfully deleted
I20240502 01:54:07.436744 21710 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:07.438278 25559 tablet_bootstrap.cc:492] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:07.450031 25001 raft_consensus.cc:1261] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:07.456583 25174 consensus_queue.cc:1035] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:07.477449 22249 raft_consensus.cc:1261] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:07.478701 25174 consensus_queue.cc:1035] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:07.492134 25592 mvcc.cc:205] Tried to move back new op lower bound from 7023062415146041344 to 7023062414207287296. Current Snapshot: MvccSnapshot[applied={T|T < 7023062415146041344}]
I20240502 01:54:07.492046 25596 tablet.cc:1722] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:07.493494 25596 tablet.cc:1620] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:07.499171 21710 catalog_manager.cc:5576] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093) does not have the latest schema for tablet a9ae84af0ef041a792f072c96644b36a (table rebalancer_dynamic_table [id=97f910a299c84acbb8c29efdc3a2a411]). Expected version 1 got 0
I20240502 01:54:07.499833 21709 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet a9ae84af0ef041a792f072c96644b36a (table rebalancer_dynamic_table [id=97f910a299c84acbb8c29efdc3a2a411]). Expected version 1 got 0
I20240502 01:54:07.500315 25594 mvcc.cc:205] Tried to move back new op lower bound from 7023062415146041344 to 7023062414207287296. Current Snapshot: MvccSnapshot[applied={T|T < 7023062415146041344}]
I20240502 01:54:07.505431 25597 tablet.cc:1722] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:07.506722 25600 tablet.cc:1722] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:07.508395 25600 tablet.cc:1620] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:07.508383 21709 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet a9ae84af0ef041a792f072c96644b36a (table rebalancer_dynamic_table [id=97f910a299c84acbb8c29efdc3a2a411]). Expected version 1 got 0
I20240502 01:54:07.516629 25597 tablet.cc:1620] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff: MemRowSet was empty: no flush needed.
I20240502 01:54:07.523768 21714 catalog_manager.cc:5576] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093) does not have the latest schema for tablet a9ae84af0ef041a792f072c96644b36a (table rebalancer_dynamic_table [id=97f910a299c84acbb8c29efdc3a2a411]). Expected version 1 got 0
I20240502 01:54:07.530431 21709 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=97f910a299c84acbb8c29efdc3a2a411] alter complete (version 1)
I20240502 01:54:07.557950 25559 tablet_bootstrap.cc:492] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:07.558493 25559 tablet_bootstrap.cc:492] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08: Bootstrap complete.
I20240502 01:54:07.558838 25559 ts_tablet_manager.cc:1367] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.121s user 0.054s sys 0.035s
I20240502 01:54:07.559613 25559 raft_consensus.cc:348] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:07.559918 25559 raft_consensus.cc:729] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.560348 25559 consensus_queue.cc:260] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:07.561172 25559 ts_tablet_manager.cc:1398] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:54:07.562284 21878 tablet_copy_service.cc:342] P 759257ea7bef415ca698e60d82199d77: Request end of tablet copy session d7449bb1df214bf4a5f93e4631312f08-071ad932ce95499fac4d7af7292b42ff received from {username='slave'} at 127.14.212.199:41477
I20240502 01:54:07.562502 21878 tablet_copy_service.cc:433] P 759257ea7bef415ca698e60d82199d77: ending tablet copy session d7449bb1df214bf4a5f93e4631312f08-071ad932ce95499fac4d7af7292b42ff on tablet 071ad932ce95499fac4d7af7292b42ff with peer d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:07.565863 24848 tablet_service.cc:1514] Processing DeleteTablet for tablet 071ad932ce95499fac4d7af7292b42ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:07 UTC) from {username='slave'} at 127.0.0.1:51592
I20240502 01:54:07.566186 25527 tablet_replica.cc:330] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08: stopping tablet replica
I20240502 01:54:07.566406 25527 raft_consensus.cc:2229] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:07.566550 25527 raft_consensus.cc:2258] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.567538 25527 ts_tablet_manager.cc:1875] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:07.573254 22118 consensus_queue.cc:237] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:07.574390 25527 ts_tablet_manager.cc:1888] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:07.574582 25527 log.cc:1196] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/wals/071ad932ce95499fac4d7af7292b42ff
I20240502 01:54:07.575086 25527 ts_tablet_manager.cc:1909] T 071ad932ce95499fac4d7af7292b42ff P d7449bb1df214bf4a5f93e4631312f08: Deleting consensus metadata
I20240502 01:54:07.576267 21700 catalog_manager.cc:4830] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): tablet 071ad932ce95499fac4d7af7292b42ff (table rebalance_test_table_18 [id=596028c83e784a669c0b5819444e54be]) successfully deleted
I20240502 01:54:07.576864 21857 raft_consensus.cc:1261] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:07.577610 25547 consensus_queue.cc:1035] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:07.577996 22250 raft_consensus.cc:1261] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:07.578593 25547 consensus_queue.cc:1035] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:07.584061 22056 consensus_peers.cc:485] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a -> Peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Couldn't send request to peer 41c70f373c9d49a49891b03a12c6cb2b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b924192d8da4a8bb7df2e6c5e4cbd4f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:07.584664 25547 raft_consensus.cc:2932] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:07.585656 22250 raft_consensus.cc:2932] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:07.592087 21857 raft_consensus.cc:2932] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:07.592928 21709 catalog_manager.cc:5484] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:07.597252 25222 raft_consensus.cc:482] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:07.597450 25222 raft_consensus.cc:504] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:07.598358 25222 leader_election.cc:288] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:07.601006 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00feee2662a48d8a1a57f9e52275154" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:07.614207 25000 raft_consensus.cc:3037] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.617602 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00feee2662a48d8a1a57f9e52275154" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:07.617893 24603 raft_consensus.cc:2379] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 41c70f373c9d49a49891b03a12c6cb2b in current term 1: Already voted for candidate 759257ea7bef415ca698e60d82199d77 in this term.
I20240502 01:54:07.618281 25409 consensus_queue.cc:1035] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:07.618412 24934 leader_election.cc:302] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, 759257ea7bef415ca698e60d82199d77
I20240502 01:54:07.618886 25222 raft_consensus.cc:2726] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:07.634756 21714 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:07.641642 25409 consensus_queue.cc:1035] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:07.646878 25596 tablet.cc:1722] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:07.647537 25596 tablet.cc:1620] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:07.647676 25600 tablet.cc:1722] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:07.648334 25600 tablet.cc:1620] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:07.648739 25322 consensus_queue.cc:1035] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:07.654731 25597 tablet.cc:1722] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:07.655328 25597 tablet.cc:1620] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff: MemRowSet was empty: no flush needed.
I20240502 01:54:07.662997 21714 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=97f910a299c84acbb8c29efdc3a2a411] alter complete (version 2)
I20240502 01:54:07.682476 25000 raft_consensus.cc:3037] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.686875 25582 consensus_queue.cc:1035] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:07.694547 22118 consensus_queue.cc:237] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:07.697424 21857 raft_consensus.cc:1261] T 166e717fb9624ae58adb6699790daf7a P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:07.698275 22250 raft_consensus.cc:1261] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:07.698316 25547 consensus_queue.cc:1035] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:07.699056 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 166e717fb9624ae58adb6699790daf7a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:07.699491 25547 consensus_queue.cc:1035] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:07.702416 25400 raft_consensus.cc:2932] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:07.705250 22250 raft_consensus.cc:2932] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:07.710332 21857 raft_consensus.cc:2932] T 166e717fb9624ae58adb6699790daf7a P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:07.710924 21714 catalog_manager.cc:5484] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:07.769135 21714 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:07.779521 25596 tablet.cc:1722] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:07.784009 25597 tablet.cc:1722] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:07.788959 21714 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=97f910a299c84acbb8c29efdc3a2a411] alter complete (version 3)
I20240502 01:54:07.789001 25600 tablet.cc:1722] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:07.809514 25589 consensus_queue.cc:1035] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:07.823657 25581 consensus_queue.cc:1035] T 96c5ae723e7a43a0b91a743b0c192a67 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:07.894924 21710 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:07.895198 21710 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:07.899096 21710 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet a9ae84af0ef041a792f072c96644b36a
I20240502 01:54:07.900213 24714 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ae84af0ef041a792f072c96644b36a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:07 UTC) from {username='slave'} at 127.0.0.1:41650
I20240502 01:54:07.900190 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ae84af0ef041a792f072c96644b36a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:07 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:07.900805 24981 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ae84af0ef041a792f072c96644b36a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:07 UTC) from {username='slave'} at 127.0.0.1:39964
I20240502 01:54:07.917088 25680 tablet_replica.cc:330] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:07.923164 25680 raft_consensus.cc:2229] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:07.924788 25680 raft_consensus.cc:2258] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.928000 25680 ts_tablet_manager.cc:1875] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:07.931665 25681 tablet_replica.cc:330] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:54:07.932049 25681 raft_consensus.cc:2229] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:07.932943 25681 raft_consensus.cc:2258] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.934026 25681 ts_tablet_manager.cc:1875] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:07.937299 25679 tablet_replica.cc:330] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:54:07.937528 25679 raft_consensus.cc:2229] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:07.937903 25679 raft_consensus.cc:2258] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.938282 25680 ts_tablet_manager.cc:1888] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:07.938460 25680 log.cc:1196] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/a9ae84af0ef041a792f072c96644b36a
I20240502 01:54:07.938979 25680 ts_tablet_manager.cc:1909] T a9ae84af0ef041a792f072c96644b36a P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:07.940418 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet a9ae84af0ef041a792f072c96644b36a (table rebalancer_dynamic_table_new_name [id=97f910a299c84acbb8c29efdc3a2a411]) successfully deleted
I20240502 01:54:07.943121 25679 ts_tablet_manager.cc:1875] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:07.952718 25679 ts_tablet_manager.cc:1888] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:07.952920 25679 log.cc:1196] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/a9ae84af0ef041a792f072c96644b36a
I20240502 01:54:07.953478 25679 ts_tablet_manager.cc:1909] T a9ae84af0ef041a792f072c96644b36a P 28157bc3e678428fb5f93c077d528513: Deleting consensus metadata
I20240502 01:54:07.954808 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet a9ae84af0ef041a792f072c96644b36a (table rebalancer_dynamic_table_new_name [id=97f910a299c84acbb8c29efdc3a2a411]) successfully deleted
I20240502 01:54:07.964623 25681 ts_tablet_manager.cc:1888] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:07.964797 25681 log.cc:1196] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/a9ae84af0ef041a792f072c96644b36a
I20240502 01:54:07.965395 25681 ts_tablet_manager.cc:1909] T a9ae84af0ef041a792f072c96644b36a P 41c70f373c9d49a49891b03a12c6cb2b: Deleting consensus metadata
I20240502 01:54:07.966542 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet a9ae84af0ef041a792f072c96644b36a (table rebalancer_dynamic_table_new_name [id=97f910a299c84acbb8c29efdc3a2a411]) successfully deleted
I20240502 01:54:07.997375 21714 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:08.004607 21714 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:08.011022 24981 tablet_service.cc:1467] Processing CreateTablet for tablet 835d8839b7d84a82befac50dcc5a7c60 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=543ab4cdef864690bedccba59308edc5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.011619 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835d8839b7d84a82befac50dcc5a7c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.011865 24848 tablet_service.cc:1467] Processing CreateTablet for tablet 59056268198e4b8f9f4d2afe1132f9c8 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=543ab4cdef864690bedccba59308edc5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.012300 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59056268198e4b8f9f4d2afe1132f9c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.017318 24847 tablet_service.cc:1467] Processing CreateTablet for tablet 835d8839b7d84a82befac50dcc5a7c60 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=543ab4cdef864690bedccba59308edc5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.017839 24847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835d8839b7d84a82befac50dcc5a7c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.021368 24714 tablet_service.cc:1467] Processing CreateTablet for tablet 59056268198e4b8f9f4d2afe1132f9c8 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=543ab4cdef864690bedccba59308edc5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.021898 24714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59056268198e4b8f9f4d2afe1132f9c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.025372 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 835d8839b7d84a82befac50dcc5a7c60 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=543ab4cdef864690bedccba59308edc5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.026080 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835d8839b7d84a82befac50dcc5a7c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.035652 24581 tablet_service.cc:1467] Processing CreateTablet for tablet 59056268198e4b8f9f4d2afe1132f9c8 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=543ab4cdef864690bedccba59308edc5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.036293 24581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59056268198e4b8f9f4d2afe1132f9c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.044209 25697 tablet_bootstrap.cc:492] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:08.056638 24846 tablet_service.cc:1467] Processing CreateTablet for tablet 61f537605bc746a0807e3c0a03ff2cc3 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=543ab4cdef864690bedccba59308edc5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.057585 24582 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:08.058810 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f537605bc746a0807e3c0a03ff2cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.059309 21967 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:08.074306 24713 tablet_service.cc:1467] Processing CreateTablet for tablet 61f537605bc746a0807e3c0a03ff2cc3 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=543ab4cdef864690bedccba59308edc5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.075039 24713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f537605bc746a0807e3c0a03ff2cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.075403 21967 tablet_service.cc:1467] Processing CreateTablet for tablet 61f537605bc746a0807e3c0a03ff2cc3 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=543ab4cdef864690bedccba59308edc5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.075917 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f537605bc746a0807e3c0a03ff2cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.099002 25697 tablet_bootstrap.cc:654] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.105854 24712 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:54:08.133433 25705 tablet_bootstrap.cc:492] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:08.144219 25705 tablet_bootstrap.cc:654] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.147495 24981 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:54:08.164577 25705 tablet_bootstrap.cc:492] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:08.165988 25705 ts_tablet_manager.cc:1367] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240502 01:54:08.167484 25705 raft_consensus.cc:348] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:08.169526 24981 tablet_service.cc:1467] Processing CreateTablet for tablet 92fb30c6561043e2a2e8c786fbeec828 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=35cd6767e16040c892ea7911c84b0189]), partition=
I20240502 01:54:08.170089 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92fb30c6561043e2a2e8c786fbeec828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.170159 25705 raft_consensus.cc:374] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.170265 25705 raft_consensus.cc:729] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.170982 24848 tablet_service.cc:1467] Processing CreateTablet for tablet 92fb30c6561043e2a2e8c786fbeec828 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=35cd6767e16040c892ea7911c84b0189]), partition=
I20240502 01:54:08.171180 25705 consensus_queue.cc:260] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:08.174487 25712 tablet_bootstrap.cc:492] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:08.177645 25705 ts_tablet_manager.cc:1398] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:54:08.185257 24582 tablet_service.cc:1467] Processing CreateTablet for tablet 92fb30c6561043e2a2e8c786fbeec828 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=35cd6767e16040c892ea7911c84b0189]), partition=
I20240502 01:54:08.185559 25712 tablet_bootstrap.cc:654] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.185756 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92fb30c6561043e2a2e8c786fbeec828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.188608 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92fb30c6561043e2a2e8c786fbeec828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.197188 25714 tablet_bootstrap.cc:492] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:08.198421 25712 tablet_bootstrap.cc:492] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:08.198693 25712 ts_tablet_manager.cc:1367] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20240502 01:54:08.200134 25714 tablet_bootstrap.cc:654] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.202116 25703 ts_tablet_manager.cc:902] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b: Initiating tablet copy from peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:08.206235 25710 ts_tablet_manager.cc:902] T 166e717fb9624ae58adb6699790daf7a P 28157bc3e678428fb5f93c077d528513: Initiating tablet copy from peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:08.206820 25709 tablet_bootstrap.cc:492] T 835d8839b7d84a82befac50dcc5a7c60 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:08.208592 25697 tablet_bootstrap.cc:492] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:08.208763 25697 ts_tablet_manager.cc:1367] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.165s user 0.002s sys 0.013s
I20240502 01:54:08.210078 25697 raft_consensus.cc:348] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:08.210407 25697 raft_consensus.cc:374] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.210635 25697 raft_consensus.cc:729] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.210942 25697 consensus_queue.cc:260] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:08.211720 25697 ts_tablet_manager.cc:1398] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:08.212551 25697 tablet_bootstrap.cc:492] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:08.206207 25712 raft_consensus.cc:348] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:08.214344 25712 raft_consensus.cc:374] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.214468 25630 raft_consensus.cc:482] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:08.214654 25630 raft_consensus.cc:504] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:08.215431 25630 leader_election.cc:288] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:08.220525 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835d8839b7d84a82befac50dcc5a7c60" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
W20240502 01:54:08.225227 24934 leader_election.cc:341] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:08.231909 25709 tablet_bootstrap.cc:654] T 835d8839b7d84a82befac50dcc5a7c60 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.232451 25703 tablet_copy_client.cc:308] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.195:38879
I20240502 01:54:08.233170 25712 raft_consensus.cc:729] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.218303 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835d8839b7d84a82befac50dcc5a7c60" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:08.233709 25712 consensus_queue.cc:260] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:08.234046 24735 raft_consensus.cc:2454] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:08.234586 24934 leader_election.cc:302] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:08.235080 25630 raft_consensus.cc:2781] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:08.235239 25630 raft_consensus.cc:482] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:08.235343 25630 raft_consensus.cc:3037] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.240034 22231 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:08.240026 25630 raft_consensus.cc:504] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:08.241233 24734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835d8839b7d84a82befac50dcc5a7c60" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:08.241490 24734 raft_consensus.cc:3037] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.249480 24734 raft_consensus.cc:2454] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:08.253151 25630 leader_election.cc:288] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:08.253082 24934 leader_election.cc:302] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:08.254065 25630 raft_consensus.cc:2781] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:08.262806 25712 ts_tablet_manager.cc:1398] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.059s user 0.002s sys 0.000s
I20240502 01:54:08.263347 25712 tablet_bootstrap.cc:492] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:08.254101 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835d8839b7d84a82befac50dcc5a7c60" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
W20240502 01:54:08.268175 24934 leader_election.cc:341] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:08.274859 24847 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:54:08.288224 25630 raft_consensus.cc:686] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:54:08.264572 25697 tablet_bootstrap.cc:654] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.288621 25630 consensus_queue.cc:237] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:08.290766 22100 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:08.305370 25710 tablet_copy_client.cc:308] T 166e717fb9624ae58adb6699790daf7a P 28157bc3e678428fb5f93c077d528513: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.195:38879
I20240502 01:54:08.306866 22140 tablet_copy_service.cc:140] P c64840657cdf4b93980bd34120c94d9a: Received BeginTabletCopySession request for tablet 166e717fb9624ae58adb6699790daf7a from peer 28157bc3e678428fb5f93c077d528513 ({username='slave'} at 127.14.212.198:48493)
I20240502 01:54:08.307148 22140 tablet_copy_service.cc:161] P c64840657cdf4b93980bd34120c94d9a: Beginning new tablet copy session on tablet 166e717fb9624ae58adb6699790daf7a from peer 28157bc3e678428fb5f93c077d528513 at {username='slave'} at 127.14.212.198:48493: session id = 28157bc3e678428fb5f93c077d528513-166e717fb9624ae58adb6699790daf7a
I20240502 01:54:08.314121 22140 tablet_copy_source_session.cc:205] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:08.317621 25710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 166e717fb9624ae58adb6699790daf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.326843 25712 tablet_bootstrap.cc:654] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.329182 25413 raft_consensus.cc:482] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:08.329433 25413 raft_consensus.cc:504] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:08.330269 25413 leader_election.cc:288] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:08.331387 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f537605bc746a0807e3c0a03ff2cc3" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:08.332319 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f537605bc746a0807e3c0a03ff2cc3" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
W20240502 01:54:08.333317 21923 leader_election.cc:341] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:54:08.333989 21923 leader_election.cc:341] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:08.334163 21923 leader_election.cc:302] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06543b05a16f42099bc86db2a0284412; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:08.335265 25413 raft_consensus.cc:2726] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:08.346838 25709 tablet_bootstrap.cc:492] T 835d8839b7d84a82befac50dcc5a7c60 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:08.347143 25709 ts_tablet_manager.cc:1367] T 835d8839b7d84a82befac50dcc5a7c60 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.140s user 0.002s sys 0.003s
I20240502 01:54:08.348172 25709 raft_consensus.cc:348] T 835d8839b7d84a82befac50dcc5a7c60 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:08.348498 25709 raft_consensus.cc:374] T 835d8839b7d84a82befac50dcc5a7c60 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.348654 25709 raft_consensus.cc:729] T 835d8839b7d84a82befac50dcc5a7c60 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.349012 25709 consensus_queue.cc:260] T 835d8839b7d84a82befac50dcc5a7c60 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:08.350296 25714 tablet_bootstrap.cc:492] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:08.350513 25714 ts_tablet_manager.cc:1367] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.153s user 0.011s sys 0.012s
I20240502 01:54:08.350653 25709 ts_tablet_manager.cc:1398] T 835d8839b7d84a82befac50dcc5a7c60 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:54:08.351357 25709 tablet_bootstrap.cc:492] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:08.351389 25714 raft_consensus.cc:348] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:08.351691 25714 raft_consensus.cc:374] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.351820 25714 raft_consensus.cc:729] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.352123 25714 consensus_queue.cc:260] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:08.352993 25714 ts_tablet_manager.cc:1398] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:08.355760 25714 tablet_bootstrap.cc:492] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:08.357677 21715 catalog_manager.cc:5484] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:08.359551 25714 tablet_bootstrap.cc:654] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.362280 25712 tablet_bootstrap.cc:492] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:08.362581 25712 ts_tablet_manager.cc:1367] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.099s user 0.005s sys 0.000s
I20240502 01:54:08.371351 25709 tablet_bootstrap.cc:654] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.371870 21836 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:08.373086 25714 tablet_bootstrap.cc:492] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:08.373358 25714 ts_tablet_manager.cc:1367] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20240502 01:54:08.374276 25714 raft_consensus.cc:348] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:08.374619 25714 raft_consensus.cc:374] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.374743 25714 raft_consensus.cc:729] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.375084 25714 consensus_queue.cc:260] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:08.375829 25714 ts_tablet_manager.cc:1398] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:08.377538 25712 raft_consensus.cc:348] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:08.377974 25712 raft_consensus.cc:374] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.378075 25712 raft_consensus.cc:729] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.381322 25712 consensus_queue.cc:260] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:08.383342 25712 ts_tablet_manager.cc:1398] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240502 01:54:08.383716 25712 tablet_bootstrap.cc:492] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:08.385555 25712 tablet_bootstrap.cc:654] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.387229 25710 tablet_copy_client.cc:783] T 166e717fb9624ae58adb6699790daf7a P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:08.387583 25712 tablet_bootstrap.cc:492] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:08.387706 25710 tablet_copy_client.cc:647] T 166e717fb9624ae58adb6699790daf7a P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:08.387770 25712 ts_tablet_manager.cc:1367] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:54:08.392993 25712 raft_consensus.cc:348] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:08.396956 22140 tablet_copy_service.cc:140] P c64840657cdf4b93980bd34120c94d9a: Received BeginTabletCopySession request for tablet 4b924192d8da4a8bb7df2e6c5e4cbd4f from peer 41c70f373c9d49a49891b03a12c6cb2b ({username='slave'} at 127.14.212.200:53727)
I20240502 01:54:08.397154 22140 tablet_copy_service.cc:161] P c64840657cdf4b93980bd34120c94d9a: Beginning new tablet copy session on tablet 4b924192d8da4a8bb7df2e6c5e4cbd4f from peer 41c70f373c9d49a49891b03a12c6cb2b at {username='slave'} at 127.14.212.200:53727: session id = 41c70f373c9d49a49891b03a12c6cb2b-4b924192d8da4a8bb7df2e6c5e4cbd4f
I20240502 01:54:08.398319 25697 tablet_bootstrap.cc:492] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:08.398444 25697 ts_tablet_manager.cc:1367] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.186s user 0.001s sys 0.004s
I20240502 01:54:08.399102 25697 raft_consensus.cc:348] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:08.399276 25697 raft_consensus.cc:374] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.399324 25697 raft_consensus.cc:729] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.399551 25697 consensus_queue.cc:260] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:08.400122 22140 tablet_copy_source_session.cc:205] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:08.400143 25697 ts_tablet_manager.cc:1398] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:08.401000 25712 raft_consensus.cc:374] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.404493 25712 raft_consensus.cc:729] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.397796 25710 tablet_copy_client.cc:523] T 166e717fb9624ae58adb6699790daf7a P 28157bc3e678428fb5f93c077d528513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:08.406512 25712 consensus_queue.cc:260] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:08.410881 25712 ts_tablet_manager.cc:1398] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.019s user 0.001s sys 0.002s
I20240502 01:54:08.412667 25709 tablet_bootstrap.cc:492] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:08.412819 25710 tablet_bootstrap.cc:492] T 166e717fb9624ae58adb6699790daf7a P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:08.413033 25709 ts_tablet_manager.cc:1367] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.000s
I20240502 01:54:08.414006 25709 raft_consensus.cc:348] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:08.414266 25709 raft_consensus.cc:374] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.414364 25709 raft_consensus.cc:729] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.414716 25709 consensus_queue.cc:260] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:08.415345 25709 ts_tablet_manager.cc:1398] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:08.415743 25709 tablet_bootstrap.cc:492] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:08.421803 25709 tablet_bootstrap.cc:654] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.426891 25703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b924192d8da4a8bb7df2e6c5e4cbd4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.433931 25703 tablet_copy_client.cc:783] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:08.434264 25703 tablet_copy_client.cc:647] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:08.437942 25703 tablet_copy_client.cc:523] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:08.438313 25582 raft_consensus.cc:482] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:08.438503 25582 raft_consensus.cc:504] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:08.439355 25582 leader_election.cc:288] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:08.440160 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59056268198e4b8f9f4d2afe1132f9c8" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:08.440426 24735 raft_consensus.cc:2454] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:08.440675 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59056268198e4b8f9f4d2afe1132f9c8" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:08.440958 24802 leader_election.cc:302] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:08.440984 24603 raft_consensus.cc:2454] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:08.442009 25582 raft_consensus.cc:2781] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:08.442124 25582 raft_consensus.cc:482] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:08.442250 25582 raft_consensus.cc:3037] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.443269 25709 tablet_bootstrap.cc:492] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:08.443419 25709 ts_tablet_manager.cc:1367] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240502 01:54:08.444267 25709 raft_consensus.cc:348] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:08.444471 25709 raft_consensus.cc:374] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.444553 25709 raft_consensus.cc:729] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.444777 25709 consensus_queue.cc:260] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:08.445399 25582 raft_consensus.cc:504] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:08.446633 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59056268198e4b8f9f4d2afe1132f9c8" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:08.446939 24735 raft_consensus.cc:3037] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.450508 24735 raft_consensus.cc:2454] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:08.451062 24802 leader_election.cc:302] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:08.452446 25582 leader_election.cc:288] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:08.454159 25582 raft_consensus.cc:2781] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:08.451867 25703 tablet_bootstrap.cc:492] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:08.454823 25582 raft_consensus.cc:686] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:08.455263 25582 consensus_queue.cc:237] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:08.478356 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59056268198e4b8f9f4d2afe1132f9c8" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:08.478911 24603 raft_consensus.cc:3037] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.483009 24603 raft_consensus.cc:2454] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:08.486507 21714 catalog_manager.cc:5484] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:08.501003 25709 ts_tablet_manager.cc:1398] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.057s user 0.007s sys 0.016s
I20240502 01:54:08.501485 25709 tablet_bootstrap.cc:492] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:08.503154 25709 tablet_bootstrap.cc:654] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.508837 25710 tablet_bootstrap.cc:492] T 166e717fb9624ae58adb6699790daf7a P 28157bc3e678428fb5f93c077d528513: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:08.509407 25710 tablet_bootstrap.cc:492] T 166e717fb9624ae58adb6699790daf7a P 28157bc3e678428fb5f93c077d528513: Bootstrap complete.
I20240502 01:54:08.509876 25710 ts_tablet_manager.cc:1367] T 166e717fb9624ae58adb6699790daf7a P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.097s user 0.054s sys 0.009s
I20240502 01:54:08.510715 25710 raft_consensus.cc:348] T 166e717fb9624ae58adb6699790daf7a P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:08.510998 25710 raft_consensus.cc:729] T 166e717fb9624ae58adb6699790daf7a P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.511267 25710 consensus_queue.cc:260] T 166e717fb9624ae58adb6699790daf7a P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:08.512221 25710 ts_tablet_manager.cc:1398] T 166e717fb9624ae58adb6699790daf7a P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:08.513727 22140 tablet_copy_service.cc:342] P c64840657cdf4b93980bd34120c94d9a: Request end of tablet copy session 28157bc3e678428fb5f93c077d528513-166e717fb9624ae58adb6699790daf7a received from {username='slave'} at 127.14.212.198:48493
I20240502 01:54:08.513926 22140 tablet_copy_service.cc:433] P c64840657cdf4b93980bd34120c94d9a: ending tablet copy session 28157bc3e678428fb5f93c077d528513-166e717fb9624ae58adb6699790daf7a on tablet 166e717fb9624ae58adb6699790daf7a with peer 28157bc3e678428fb5f93c077d528513
I20240502 01:54:08.538408 25709 tablet_bootstrap.cc:492] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:08.539355 25709 ts_tablet_manager.cc:1367] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.014s
I20240502 01:54:08.539965 24735 raft_consensus.cc:1203] T 166e717fb9624ae58adb6699790daf7a P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:08.540309 25709 raft_consensus.cc:348] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:08.540551 25709 raft_consensus.cc:374] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.540645 25709 raft_consensus.cc:729] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.540957 25709 consensus_queue.cc:260] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:08.541872 25709 ts_tablet_manager.cc:1398] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:08.557416 25703 tablet_bootstrap.cc:492] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:08.558050 25703 tablet_bootstrap.cc:492] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap complete.
I20240502 01:54:08.558458 25703 ts_tablet_manager.cc:1367] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.107s user 0.063s sys 0.001s
I20240502 01:54:08.559401 25703 raft_consensus.cc:348] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:08.559726 25703 raft_consensus.cc:729] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.560024 25703 consensus_queue.cc:260] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:08.560693 25703 ts_tablet_manager.cc:1398] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:08.562026 22140 tablet_copy_service.cc:342] P c64840657cdf4b93980bd34120c94d9a: Request end of tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-4b924192d8da4a8bb7df2e6c5e4cbd4f received from {username='slave'} at 127.14.212.200:53727
I20240502 01:54:08.562187 22140 tablet_copy_service.cc:433] P c64840657cdf4b93980bd34120c94d9a: ending tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-4b924192d8da4a8bb7df2e6c5e4cbd4f on tablet 4b924192d8da4a8bb7df2e6c5e4cbd4f with peer 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:08.615355 22118 consensus_queue.cc:237] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:08.618341 25307 raft_consensus.cc:482] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:08.618340 21988 raft_consensus.cc:1261] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:08.618544 25307 raft_consensus.cc:504] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:08.619050 25400 consensus_queue.cc:1035] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:08.619261 25307 leader_election.cc:288] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:08.620147 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb30c6561043e2a2e8c786fbeec828" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
W20240502 01:54:08.620349 22054 consensus_peers.cc:485] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a -> Peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Couldn't send request to peer 0c64d40b5d7644bd8b29feaa80eab72f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3cf315f0117444ae9c5390a1e19460d9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:08.620465 24867 raft_consensus.cc:2454] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:08.621001 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb30c6561043e2a2e8c786fbeec828" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:08.621268 25000 raft_consensus.cc:2454] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:08.621273 21857 raft_consensus.cc:1261] T 3cf315f0117444ae9c5390a1e19460d9 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:08.621807 24539 leader_election.cc:302] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:08.622125 25307 raft_consensus.cc:2781] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:08.622238 25307 raft_consensus.cc:482] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:08.622346 25307 raft_consensus.cc:3037] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.622905 25547 consensus_queue.cc:1035] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:08.624816 25307 raft_consensus.cc:504] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:08.626302 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb30c6561043e2a2e8c786fbeec828" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:08.626523 25000 raft_consensus.cc:3037] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.628466 25000 raft_consensus.cc:2454] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:08.629001 24539 leader_election.cc:302] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:08.629539 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fb30c6561043e2a2e8c786fbeec828" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:08.629729 25763 raft_consensus.cc:2781] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:08.629784 24867 raft_consensus.cc:3037] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.630324 25763 raft_consensus.cc:686] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:54:08.630766 25763 consensus_queue.cc:237] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:08.632057 24867 raft_consensus.cc:2454] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:08.632357 25739 raft_consensus.cc:482] T 835d8839b7d84a82befac50dcc5a7c60 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:08.632370 25307 leader_election.cc:288] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:08.632519 25739 raft_consensus.cc:504] T 835d8839b7d84a82befac50dcc5a7c60 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:08.633328 25739 leader_election.cc:288] T 835d8839b7d84a82befac50dcc5a7c60 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:08.634408 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835d8839b7d84a82befac50dcc5a7c60" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:08.634725 24735 raft_consensus.cc:2379] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7449bb1df214bf4a5f93e4631312f08 in current term 1: Already voted for candidate 41c70f373c9d49a49891b03a12c6cb2b in this term.
I20240502 01:54:08.635318 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835d8839b7d84a82befac50dcc5a7c60" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:08.635895 24804 leader_election.cc:302] T 835d8839b7d84a82befac50dcc5a7c60 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7449bb1df214bf4a5f93e4631312f08; no voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:08.636183 25739 raft_consensus.cc:3037] T 835d8839b7d84a82befac50dcc5a7c60 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.638985 25739 raft_consensus.cc:2726] T 835d8839b7d84a82befac50dcc5a7c60 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:08.639560 25739 raft_consensus.cc:482] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:08.639787 25739 raft_consensus.cc:504] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:08.640120 25760 raft_consensus.cc:2932] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:08.641554 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f537605bc746a0807e3c0a03ff2cc3" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:08.641844 24735 raft_consensus.cc:2454] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:08.641932 21710 catalog_manager.cc:5484] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:08.642411 24802 leader_election.cc:302] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:08.642709 25739 leader_election.cc:288] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:08.642741 25738 raft_consensus.cc:2781] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:08.642903 25738 raft_consensus.cc:482] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:08.642995 25738 raft_consensus.cc:3037] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.645884 25738 raft_consensus.cc:504] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:08.645658 21987 raft_consensus.cc:2932] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:08.647022 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f537605bc746a0807e3c0a03ff2cc3" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:54:08.647248 21989 raft_consensus.cc:3037] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.646903 21854 raft_consensus.cc:2932] T 3cf315f0117444ae9c5390a1e19460d9 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:08.647450 25738 leader_election.cc:288] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:08.649394 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f537605bc746a0807e3c0a03ff2cc3" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:08.649645 24735 raft_consensus.cc:3037] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.651999 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f537605bc746a0807e3c0a03ff2cc3" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:54:08.652333 24735 raft_consensus.cc:2454] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:08.653165 24802 leader_election.cc:302] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:08.653710 25582 raft_consensus.cc:2781] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:08.653843 25582 raft_consensus.cc:686] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:08.654126 25582 consensus_queue.cc:237] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:08.657334 21989 raft_consensus.cc:2454] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:08.658234 21714 catalog_manager.cc:5484] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:08.660923 25000 raft_consensus.cc:1203] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:08.664384 21714 catalog_manager.cc:5484] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:08.706629 25630 consensus_queue.cc:1035] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:08.731196 25630 consensus_queue.cc:1035] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:08.748579 21989 consensus_queue.cc:237] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
W20240502 01:54:08.751613 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db96ccbe81564bb589b0de65977b3321. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:08.751665 22118 raft_consensus.cc:1261] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:08.751956 21854 raft_consensus.cc:1261] T db96ccbe81564bb589b0de65977b3321 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:08.752563 25791 consensus_queue.cc:1035] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:08.752873 25464 consensus_queue.cc:1035] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:08.762130 25791 raft_consensus.cc:2932] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:08.762812 22117 raft_consensus.cc:2932] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:08.766829 21854 raft_consensus.cc:2932] T db96ccbe81564bb589b0de65977b3321 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:08.769245 21714 catalog_manager.cc:5484] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:08.808712 21854 consensus_queue.cc:237] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:08.811981 22249 raft_consensus.cc:1261] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:08.812564 25409 consensus_queue.cc:1035] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:08.813220 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d52f4b5c815f4188a09eb12ec5375361. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:08.814209 22117 raft_consensus.cc:1261] T d52f4b5c815f4188a09eb12ec5375361 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:08.816118 25409 consensus_queue.cc:1035] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:08.819176 25409 raft_consensus.cc:2932] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:08.819671 22117 raft_consensus.cc:2932] T d52f4b5c815f4188a09eb12ec5375361 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:08.824283 21710 catalog_manager.cc:5484] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:08.827214 22249 raft_consensus.cc:2932] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:08.837481 21714 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:08.846798 24867 raft_consensus.cc:1261] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:08.847559 25763 consensus_queue.cc:1035] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:08.847869 25000 raft_consensus.cc:1261] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:08.850566 25763 consensus_queue.cc:1035] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:08.855367 21714 catalog_manager.cc:5576] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737) does not have the latest schema for tablet 92fb30c6561043e2a2e8c786fbeec828 (table rebalancer_dynamic_table [id=35cd6767e16040c892ea7911c84b0189]). Expected version 1 got 0
I20240502 01:54:08.858915 21714 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 92fb30c6561043e2a2e8c786fbeec828 (table rebalancer_dynamic_table [id=35cd6767e16040c892ea7911c84b0189]). Expected version 1 got 0
I20240502 01:54:08.860966 25821 tablet.cc:1722] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:08.861708 25817 mvcc.cc:205] Tried to move back new op lower bound from 7023062420869496832 to 7023062419995553792. Current Snapshot: MvccSnapshot[applied={T|T < 7023062420869496832}]
I20240502 01:54:08.863152 25821 tablet.cc:1620] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:08.869874 25822 tablet.cc:1722] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:08.871467 25822 tablet.cc:1620] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:08.874665 21714 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 92fb30c6561043e2a2e8c786fbeec828 (table rebalancer_dynamic_table [id=35cd6767e16040c892ea7911c84b0189]). Expected version 1 got 0
I20240502 01:54:08.877642 25820 tablet.cc:1722] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:08.878280 25820 tablet.cc:1620] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:08.881017 21710 catalog_manager.cc:5576] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737) does not have the latest schema for tablet 92fb30c6561043e2a2e8c786fbeec828 (table rebalancer_dynamic_table [id=35cd6767e16040c892ea7911c84b0189]). Expected version 1 got 0
I20240502 01:54:08.899204 21714 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=35cd6767e16040c892ea7911c84b0189] alter complete (version 1)
I20240502 01:54:08.936340 22117 consensus_queue.cc:237] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:08.938771 21854 raft_consensus.cc:1261] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:08.939405 22249 raft_consensus.cc:1261] T ad3cd6fff977431b9cd5df6d083bef80 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:08.939512 22056 consensus_peers.cc:485] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a -> Peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Couldn't send request to peer 41c70f373c9d49a49891b03a12c6cb2b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad3cd6fff977431b9cd5df6d083bef80. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:08.939538 25760 consensus_queue.cc:1035] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:08.940080 25766 consensus_queue.cc:1035] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:08.942834 25766 raft_consensus.cc:2932] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:08.943423 21854 raft_consensus.cc:2932] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:08.947178 22249 raft_consensus.cc:2932] T ad3cd6fff977431b9cd5df6d083bef80 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:08.948566 21714 catalog_manager.cc:5484] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:08.982249 25000 consensus_queue.cc:237] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:08.984779 24867 raft_consensus.cc:1261] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:08.985451 21854 raft_consensus.cc:1261] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:08.985535 25726 consensus_queue.cc:1035] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:08.986055 25787 consensus_queue.cc:1035] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:08.989977 25726 raft_consensus.cc:2932] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:08.990774 24867 raft_consensus.cc:2932] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:08.993012 25582 consensus_queue.cc:1035] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:08.994028 24933 consensus_peers.cc:485] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 992544bbea6547de96298fd32d74b232. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:08.995643 21854 raft_consensus.cc:2932] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:08.996811 21715 catalog_manager.cc:5484] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:09.004508 21714 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:09.006798 25738 consensus_queue.cc:1035] T 59056268198e4b8f9f4d2afe1132f9c8 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.019851 25821 tablet.cc:1722] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:09.020448 25821 tablet.cc:1620] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:09.020869 25822 tablet.cc:1722] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:09.021116 25820 tablet.cc:1722] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:09.021328 25822 tablet.cc:1620] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:09.021786 25820 tablet.cc:1620] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:09.036180 21715 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=35cd6767e16040c892ea7911c84b0189] alter complete (version 2)
I20240502 01:54:09.066102 24867 consensus_queue.cc:237] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:09.068769 24735 raft_consensus.cc:1261] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:09.068827 24603 raft_consensus.cc:1261] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:09.069806 25582 consensus_queue.cc:1035] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:09.070219 25738 consensus_queue.cc:1035] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:09.073488 25582 raft_consensus.cc:2932] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:09.074067 24603 raft_consensus.cc:2932] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:09.080238 21710 catalog_manager.cc:5484] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
W20240502 01:54:09.081239 24803 consensus_peers.cc:485] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b60b948dae4841dd9c0919f2fcac5299. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:09.082338 24735 raft_consensus.cc:2932] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:09.118064 22118 consensus_queue.cc:237] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:09.120481 21854 raft_consensus.cc:1261] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:09.120673 22249 raft_consensus.cc:1261] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:09.120769 22056 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Couldn't send request to peer 41c70f373c9d49a49891b03a12c6cb2b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: daaadb773202454894890c3379c4a58d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:09.121148 25400 consensus_queue.cc:1035] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:09.121443 25760 consensus_queue.cc:1035] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:09.124351 25400 raft_consensus.cc:2932] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:09.125046 21854 raft_consensus.cc:2932] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:09.131475 22249 raft_consensus.cc:2932] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:09.131808 21714 catalog_manager.cc:5484] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:09.141438 21714 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:09.150389 25821 tablet.cc:1722] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:09.150998 25822 tablet.cc:1722] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:09.151274 25820 tablet.cc:1722] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:09.161300 21715 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=35cd6767e16040c892ea7911c84b0189] alter complete (version 3)
I20240502 01:54:09.169785 25910 ts_tablet_manager.cc:902] T 3cf315f0117444ae9c5390a1e19460d9 P 0c64d40b5d7644bd8b29feaa80eab72f: Initiating tablet copy from peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:09.171808 25910 tablet_copy_client.cc:308] T 3cf315f0117444ae9c5390a1e19460d9 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.195:38879
I20240502 01:54:09.172822 22140 tablet_copy_service.cc:140] P c64840657cdf4b93980bd34120c94d9a: Received BeginTabletCopySession request for tablet 3cf315f0117444ae9c5390a1e19460d9 from peer 0c64d40b5d7644bd8b29feaa80eab72f ({username='slave'} at 127.14.212.197:58377)
I20240502 01:54:09.173038 22140 tablet_copy_service.cc:161] P c64840657cdf4b93980bd34120c94d9a: Beginning new tablet copy session on tablet 3cf315f0117444ae9c5390a1e19460d9 from peer 0c64d40b5d7644bd8b29feaa80eab72f at {username='slave'} at 127.14.212.197:58377: session id = 0c64d40b5d7644bd8b29feaa80eab72f-3cf315f0117444ae9c5390a1e19460d9
I20240502 01:54:09.176018 22140 tablet_copy_source_session.cc:205] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:09.178004 25910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf315f0117444ae9c5390a1e19460d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.184168 25910 tablet_copy_client.cc:783] T 3cf315f0117444ae9c5390a1e19460d9 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:09.185045 25910 tablet_copy_client.cc:647] T 3cf315f0117444ae9c5390a1e19460d9 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:09.187440 25910 tablet_copy_client.cc:523] T 3cf315f0117444ae9c5390a1e19460d9 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:09.190515 25910 tablet_bootstrap.cc:492] T 3cf315f0117444ae9c5390a1e19460d9 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:09.191164 24735 consensus_queue.cc:237] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:09.194159 24603 raft_consensus.cc:1261] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:09.194983 24668 consensus_peers.cc:485] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03fb607ef0d9469daca1d6f3c5259fb2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:09.194983 25581 consensus_queue.cc:1035] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:09.198058 21988 raft_consensus.cc:1261] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:09.198621 25581 consensus_queue.cc:1035] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:09.206720 25582 consensus_queue.cc:1035] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.212396 25738 consensus_queue.cc:1035] T 61f537605bc746a0807e3c0a03ff2cc3 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.214783 25581 raft_consensus.cc:2932] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:09.215510 21710 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:09.219238 24601 raft_consensus.cc:2932] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:09.226644 21715 catalog_manager.cc:5484] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:09.229950 21989 raft_consensus.cc:2932] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:09.237465 24713 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab9f7fe67aa41b5971d5df9e4b5d788 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=091ceb1644c44e13b8f3ed4e93da3341]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.237980 24713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab9f7fe67aa41b5971d5df9e4b5d788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.241267 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 37b77f3edaf3431a96551a9525197579 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=091ceb1644c44e13b8f3ed4e93da3341]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.241556 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b77f3edaf3431a96551a9525197579. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.245381 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab9f7fe67aa41b5971d5df9e4b5d788 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=091ceb1644c44e13b8f3ed4e93da3341]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.245901 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab9f7fe67aa41b5971d5df9e4b5d788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.249449 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 05bb6b1075f442aaa3bbe005c98ce610 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=091ceb1644c44e13b8f3ed4e93da3341]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.250118 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05bb6b1075f442aaa3bbe005c98ce610. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.253711 24581 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab9f7fe67aa41b5971d5df9e4b5d788 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=091ceb1644c44e13b8f3ed4e93da3341]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.254209 24581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab9f7fe67aa41b5971d5df9e4b5d788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.258997 24848 tablet_service.cc:1467] Processing CreateTablet for tablet 37b77f3edaf3431a96551a9525197579 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=091ceb1644c44e13b8f3ed4e93da3341]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.259343 24980 tablet_service.cc:1467] Processing CreateTablet for tablet 05bb6b1075f442aaa3bbe005c98ce610 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=091ceb1644c44e13b8f3ed4e93da3341]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.259346 24981 tablet_service.cc:1467] Processing CreateTablet for tablet 37b77f3edaf3431a96551a9525197579 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=091ceb1644c44e13b8f3ed4e93da3341]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.259583 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b77f3edaf3431a96551a9525197579. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.259856 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b77f3edaf3431a96551a9525197579. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.262428 24847 tablet_service.cc:1467] Processing CreateTablet for tablet 05bb6b1075f442aaa3bbe005c98ce610 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=091ceb1644c44e13b8f3ed4e93da3341]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.262681 24980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05bb6b1075f442aaa3bbe005c98ce610. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.262822 24847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05bb6b1075f442aaa3bbe005c98ce610. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.267957 21710 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:09.268213 21710 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:09.274329 21710 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 92fb30c6561043e2a2e8c786fbeec828
I20240502 01:54:09.275616 24979 tablet_service.cc:1514] Processing DeleteTablet for tablet 92fb30c6561043e2a2e8c786fbeec828 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:09 UTC) from {username='slave'} at 127.0.0.1:39964
I20240502 01:54:09.275951 24846 tablet_service.cc:1514] Processing DeleteTablet for tablet 92fb30c6561043e2a2e8c786fbeec828 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:09 UTC) from {username='slave'} at 127.0.0.1:51592
I20240502 01:54:09.276227 24582 tablet_service.cc:1514] Processing DeleteTablet for tablet 92fb30c6561043e2a2e8c786fbeec828 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:09 UTC) from {username='slave'} at 127.0.0.1:53832
I20240502 01:54:09.296943 25936 tablet_bootstrap.cc:492] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:09.305887 25933 tablet_replica.cc:330] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08: stopping tablet replica
I20240502 01:54:09.306247 25933 raft_consensus.cc:2229] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:09.306602 25933 raft_consensus.cc:2258] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:09.306771 25936 tablet_bootstrap.cc:654] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.307719 25933 ts_tablet_manager.cc:1875] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:09.313311 25935 tablet_bootstrap.cc:492] T 37b77f3edaf3431a96551a9525197579 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:09.315891 25935 tablet_bootstrap.cc:654] T 37b77f3edaf3431a96551a9525197579 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.331182 25939 tablet_bootstrap.cc:492] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:09.333786 25939 tablet_bootstrap.cc:654] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.338169 25938 tablet_bootstrap.cc:492] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:09.342070 25937 tablet_bootstrap.cc:492] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:09.346042 25932 tablet_replica.cc:330] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:54:09.346396 25932 raft_consensus.cc:2229] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:09.346751 25932 raft_consensus.cc:2258] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:09.347757 25938 tablet_bootstrap.cc:654] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.347961 25932 ts_tablet_manager.cc:1875] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:09.348639 25937 tablet_bootstrap.cc:654] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.350739 25931 ts_tablet_manager.cc:902] T d52f4b5c815f4188a09eb12ec5375361 P 28157bc3e678428fb5f93c077d528513: Initiating tablet copy from peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:09.351871 25934 tablet_replica.cc:330] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:54:09.352105 25936 tablet_bootstrap.cc:492] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:09.352329 25936 ts_tablet_manager.cc:1367] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.034s
I20240502 01:54:09.352623 25934 raft_consensus.cc:2229] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:09.353175 25936 raft_consensus.cc:348] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.353446 25936 raft_consensus.cc:374] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.353493 25934 raft_consensus.cc:2258] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:09.353574 25937 tablet_bootstrap.cc:492] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:09.353566 25936 raft_consensus.cc:729] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.353760 25937 ts_tablet_manager.cc:1367] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20240502 01:54:09.353839 25936 consensus_queue.cc:260] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.354439 25941 ts_tablet_manager.cc:902] T db96ccbe81564bb589b0de65977b3321 P d7449bb1df214bf4a5f93e4631312f08: Initiating tablet copy from peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:09.358000 25936 ts_tablet_manager.cc:1398] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:54:09.358341 25935 tablet_bootstrap.cc:492] T 37b77f3edaf3431a96551a9525197579 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:09.360478 25937 raft_consensus.cc:348] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:09.360670 25935 ts_tablet_manager.cc:1367] T 37b77f3edaf3431a96551a9525197579 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.000s
I20240502 01:54:09.361677 25937 raft_consensus.cc:374] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.361652 25935 raft_consensus.cc:348] T 37b77f3edaf3431a96551a9525197579 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.361910 25935 raft_consensus.cc:374] T 37b77f3edaf3431a96551a9525197579 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.362039 25935 raft_consensus.cc:729] T 37b77f3edaf3431a96551a9525197579 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.362416 25937 raft_consensus.cc:729] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.360257 25933 ts_tablet_manager.cc:1888] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:09.363359 25937 consensus_queue.cc:260] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:09.363708 25933 log.cc:1196] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/wals/92fb30c6561043e2a2e8c786fbeec828
I20240502 01:54:09.364282 25933 ts_tablet_manager.cc:1909] T 92fb30c6561043e2a2e8c786fbeec828 P d7449bb1df214bf4a5f93e4631312f08: Deleting consensus metadata
I20240502 01:54:09.364267 25934 ts_tablet_manager.cc:1875] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:09.365881 21700 catalog_manager.cc:4830] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): tablet 92fb30c6561043e2a2e8c786fbeec828 (table rebalancer_dynamic_table_new_name [id=35cd6767e16040c892ea7911c84b0189]) successfully deleted
I20240502 01:54:09.366551 25937 ts_tablet_manager.cc:1398] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.013s user 0.000s sys 0.003s
I20240502 01:54:09.367229 25938 tablet_bootstrap.cc:492] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:09.367605 25938 ts_tablet_manager.cc:1367] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.000s
I20240502 01:54:09.364557 25935 consensus_queue.cc:260] T 37b77f3edaf3431a96551a9525197579 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.368325 25930 tablet_bootstrap.cc:492] T 37b77f3edaf3431a96551a9525197579 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:09.368649 25938 raft_consensus.cc:348] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:09.368959 25938 raft_consensus.cc:374] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.369107 25938 raft_consensus.cc:729] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.370723 25939 tablet_bootstrap.cc:492] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:09.370975 25939 ts_tablet_manager.cc:1367] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.004s
I20240502 01:54:09.371474 25935 ts_tablet_manager.cc:1398] T 37b77f3edaf3431a96551a9525197579 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:54:09.372001 25939 raft_consensus.cc:348] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:09.372279 25939 raft_consensus.cc:374] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.372413 25939 raft_consensus.cc:729] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.372727 25939 consensus_queue.cc:260] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:09.373533 25939 ts_tablet_manager.cc:1398] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:09.373783 25938 consensus_queue.cc:260] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:09.374734 25935 tablet_bootstrap.cc:492] T 05bb6b1075f442aaa3bbe005c98ce610 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:09.376055 25938 ts_tablet_manager.cc:1398] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:54:09.376662 21709 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:09.385439 25935 tablet_bootstrap.cc:654] T 05bb6b1075f442aaa3bbe005c98ce610 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.387967 25930 tablet_bootstrap.cc:654] T 37b77f3edaf3431a96551a9525197579 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.389087 25286 raft_consensus.cc:482] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:09.389289 25286 raft_consensus.cc:504] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:09.390266 25286 leader_election.cc:288] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:09.394697 24846 tablet_service.cc:1467] Processing CreateTablet for tablet 9850769ab02d497891665afb11679201 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ffb30602844e4d1fb3265fad1e97a6ae]), partition=
I20240502 01:54:09.395198 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9850769ab02d497891665afb11679201. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.399667 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05bb6b1075f442aaa3bbe005c98ce610" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
W20240502 01:54:09.400353 22185 leader_election.cc:341] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:09.405587 24979 tablet_service.cc:1467] Processing CreateTablet for tablet 9850769ab02d497891665afb11679201 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ffb30602844e4d1fb3265fad1e97a6ae]), partition=
I20240502 01:54:09.406427 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9850769ab02d497891665afb11679201. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.413290 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 9850769ab02d497891665afb11679201 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ffb30602844e4d1fb3265fad1e97a6ae]), partition=
I20240502 01:54:09.413884 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9850769ab02d497891665afb11679201. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.419350 25932 ts_tablet_manager.cc:1888] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:09.419517 25932 log.cc:1196] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/92fb30c6561043e2a2e8c786fbeec828
I20240502 01:54:09.420156 25932 ts_tablet_manager.cc:1909] T 92fb30c6561043e2a2e8c786fbeec828 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting consensus metadata
I20240502 01:54:09.421645 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet 92fb30c6561043e2a2e8c786fbeec828 (table rebalancer_dynamic_table_new_name [id=35cd6767e16040c892ea7911c84b0189]) successfully deleted
I20240502 01:54:09.422207 25936 tablet_bootstrap.cc:492] T 05bb6b1075f442aaa3bbe005c98ce610 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:09.424573 25936 tablet_bootstrap.cc:654] T 05bb6b1075f442aaa3bbe005c98ce610 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.435357 25957 ts_tablet_manager.cc:902] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b: Initiating tablet copy from peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:09.437394 25931 tablet_copy_client.cc:308] T d52f4b5c815f4188a09eb12ec5375361 P 28157bc3e678428fb5f93c077d528513: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.193:33785
I20240502 01:54:09.437770 25787 raft_consensus.cc:482] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:09.437955 25787 raft_consensus.cc:504] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.438213 21878 tablet_copy_service.cc:140] P 759257ea7bef415ca698e60d82199d77: Received BeginTabletCopySession request for tablet d52f4b5c815f4188a09eb12ec5375361 from peer 28157bc3e678428fb5f93c077d528513 ({username='slave'} at 127.14.212.198:60613)
I20240502 01:54:09.438406 21878 tablet_copy_service.cc:161] P 759257ea7bef415ca698e60d82199d77: Beginning new tablet copy session on tablet d52f4b5c815f4188a09eb12ec5375361 from peer 28157bc3e678428fb5f93c077d528513 at {username='slave'} at 127.14.212.198:60613: session id = 28157bc3e678428fb5f93c077d528513-d52f4b5c815f4188a09eb12ec5375361
I20240502 01:54:09.438736 25787 leader_election.cc:288] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:09.438791 25934 ts_tablet_manager.cc:1888] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:09.438958 25934 log.cc:1196] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/92fb30c6561043e2a2e8c786fbeec828
I20240502 01:54:09.439543 25934 ts_tablet_manager.cc:1909] T 92fb30c6561043e2a2e8c786fbeec828 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting consensus metadata
I20240502 01:54:09.440613 25930 tablet_bootstrap.cc:492] T 37b77f3edaf3431a96551a9525197579 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:09.440822 25930 ts_tablet_manager.cc:1367] T 37b77f3edaf3431a96551a9525197579 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.073s user 0.008s sys 0.000s
I20240502 01:54:09.441424 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet 92fb30c6561043e2a2e8c786fbeec828 (table rebalancer_dynamic_table_new_name [id=35cd6767e16040c892ea7911c84b0189]) successfully deleted
I20240502 01:54:09.441680 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b77f3edaf3431a96551a9525197579" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:09.441933 25941 tablet_copy_client.cc:308] T db96ccbe81564bb589b0de65977b3321 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:44411
I20240502 01:54:09.441995 24867 raft_consensus.cc:2454] T 37b77f3edaf3431a96551a9525197579 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:09.442065 25930 raft_consensus.cc:348] T 37b77f3edaf3431a96551a9525197579 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.442292 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b77f3edaf3431a96551a9525197579" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:09.442672 24934 leader_election.cc:302] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:09.442801 25930 raft_consensus.cc:374] T 37b77f3edaf3431a96551a9525197579 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.442921 25930 raft_consensus.cc:729] T 37b77f3edaf3431a96551a9525197579 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.442957 21878 tablet_copy_source_session.cc:205] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:09.443087 25787 raft_consensus.cc:2781] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:09.443187 25787 raft_consensus.cc:482] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:09.443298 25787 raft_consensus.cc:3037] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.443327 25930 consensus_queue.cc:260] T 37b77f3edaf3431a96551a9525197579 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.443820 24735 raft_consensus.cc:2454] T 37b77f3edaf3431a96551a9525197579 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:09.446272 25930 ts_tablet_manager.cc:1398] T 37b77f3edaf3431a96551a9525197579 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:09.446194 25787 raft_consensus.cc:504] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.448091 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b77f3edaf3431a96551a9525197579" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:09.448326 24867 raft_consensus.cc:3037] T 37b77f3edaf3431a96551a9525197579 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.449990 25935 tablet_bootstrap.cc:492] T 05bb6b1075f442aaa3bbe005c98ce610 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:09.450193 25935 ts_tablet_manager.cc:1367] T 05bb6b1075f442aaa3bbe005c98ce610 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.075s user 0.002s sys 0.010s
I20240502 01:54:09.451110 25935 raft_consensus.cc:348] T 05bb6b1075f442aaa3bbe005c98ce610 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:09.451324 25935 raft_consensus.cc:374] T 05bb6b1075f442aaa3bbe005c98ce610 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.451411 24867 raft_consensus.cc:2454] T 37b77f3edaf3431a96551a9525197579 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:09.451414 25935 raft_consensus.cc:729] T 05bb6b1075f442aaa3bbe005c98ce610 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.451934 24934 leader_election.cc:302] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:09.451906 25935 consensus_queue.cc:260] T 05bb6b1075f442aaa3bbe005c98ce610 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:09.452275 25726 raft_consensus.cc:2781] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:09.452404 25726 raft_consensus.cc:686] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:54:09.452693 25726 consensus_queue.cc:237] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.453500 25935 ts_tablet_manager.cc:1398] T 05bb6b1075f442aaa3bbe005c98ce610 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 01:54:09.453797 25935 tablet_bootstrap.cc:492] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:09.455195 25787 leader_election.cc:288] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:09.456677 25935 tablet_bootstrap.cc:654] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.457089 25936 tablet_bootstrap.cc:492] T 05bb6b1075f442aaa3bbe005c98ce610 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:09.457392 25936 ts_tablet_manager.cc:1367] T 05bb6b1075f442aaa3bbe005c98ce610 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.002s
I20240502 01:54:09.461252 25936 raft_consensus.cc:348] T 05bb6b1075f442aaa3bbe005c98ce610 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:09.461463 22010 tablet_copy_service.cc:140] P 06543b05a16f42099bc86db2a0284412: Received BeginTabletCopySession request for tablet db96ccbe81564bb589b0de65977b3321 from peer d7449bb1df214bf4a5f93e4631312f08 ({username='slave'} at 127.14.212.199:39887)
I20240502 01:54:09.461599 25936 raft_consensus.cc:374] T 05bb6b1075f442aaa3bbe005c98ce610 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.461748 22010 tablet_copy_service.cc:161] P 06543b05a16f42099bc86db2a0284412: Beginning new tablet copy session on tablet db96ccbe81564bb589b0de65977b3321 from peer d7449bb1df214bf4a5f93e4631312f08 at {username='slave'} at 127.14.212.199:39887: session id = d7449bb1df214bf4a5f93e4631312f08-db96ccbe81564bb589b0de65977b3321
I20240502 01:54:09.462548 25936 raft_consensus.cc:729] T 05bb6b1075f442aaa3bbe005c98ce610 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.462926 25936 consensus_queue.cc:260] T 05bb6b1075f442aaa3bbe005c98ce610 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:09.463753 25936 ts_tablet_manager.cc:1398] T 05bb6b1075f442aaa3bbe005c98ce610 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:54:09.466023 25930 tablet_bootstrap.cc:492] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:09.466523 25936 tablet_bootstrap.cc:492] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:09.447348 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b77f3edaf3431a96551a9525197579" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:09.466795 24735 raft_consensus.cc:3037] T 37b77f3edaf3431a96551a9525197579 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.468567 25935 tablet_bootstrap.cc:492] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:09.468786 25935 ts_tablet_manager.cc:1367] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240502 01:54:09.469669 25935 raft_consensus.cc:348] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.469911 25935 raft_consensus.cc:374] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.470023 25935 raft_consensus.cc:729] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.470309 25935 consensus_queue.cc:260] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.471362 25936 tablet_bootstrap.cc:654] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.471768 25935 ts_tablet_manager.cc:1398] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:09.469689 25957 tablet_copy_client.cc:308] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.195:38879
I20240502 01:54:09.473193 22140 tablet_copy_service.cc:140] P c64840657cdf4b93980bd34120c94d9a: Received BeginTabletCopySession request for tablet ad3cd6fff977431b9cd5df6d083bef80 from peer 41c70f373c9d49a49891b03a12c6cb2b ({username='slave'} at 127.14.212.200:53727)
I20240502 01:54:09.473237 24735 raft_consensus.cc:2454] T 37b77f3edaf3431a96551a9525197579 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:09.473402 22140 tablet_copy_service.cc:161] P c64840657cdf4b93980bd34120c94d9a: Beginning new tablet copy session on tablet ad3cd6fff977431b9cd5df6d083bef80 from peer 41c70f373c9d49a49891b03a12c6cb2b at {username='slave'} at 127.14.212.200:53727: session id = 41c70f373c9d49a49891b03a12c6cb2b-ad3cd6fff977431b9cd5df6d083bef80
I20240502 01:54:09.474510 25931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52f4b5c815f4188a09eb12ec5375361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.475636 22010 tablet_copy_source_session.cc:205] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:09.476450 22140 tablet_copy_source_session.cc:205] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:09.478296 25910 tablet_bootstrap.cc:492] T 3cf315f0117444ae9c5390a1e19460d9 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:09.478391 25957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3cd6fff977431b9cd5df6d083bef80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.481976 25941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db96ccbe81564bb589b0de65977b3321. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.485670 25930 tablet_bootstrap.cc:654] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.488313 25910 tablet_bootstrap.cc:492] T 3cf315f0117444ae9c5390a1e19460d9 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap complete.
I20240502 01:54:09.488734 25910 ts_tablet_manager.cc:1367] T 3cf315f0117444ae9c5390a1e19460d9 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.298s user 0.062s sys 0.006s
I20240502 01:54:09.489750 25910 raft_consensus.cc:348] T 3cf315f0117444ae9c5390a1e19460d9 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:09.490154 25910 raft_consensus.cc:729] T 3cf315f0117444ae9c5390a1e19460d9 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.490484 25910 consensus_queue.cc:260] T 3cf315f0117444ae9c5390a1e19460d9 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:09.492005 25910 ts_tablet_manager.cc:1398] T 3cf315f0117444ae9c5390a1e19460d9 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:09.493045 22140 tablet_copy_service.cc:342] P c64840657cdf4b93980bd34120c94d9a: Request end of tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-3cf315f0117444ae9c5390a1e19460d9 received from {username='slave'} at 127.14.212.197:58377
I20240502 01:54:09.493165 22140 tablet_copy_service.cc:433] P c64840657cdf4b93980bd34120c94d9a: ending tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-3cf315f0117444ae9c5390a1e19460d9 on tablet 3cf315f0117444ae9c5390a1e19460d9 with peer 0c64d40b5d7644bd8b29feaa80eab72f
I20240502 01:54:09.494817 21715 catalog_manager.cc:5484] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:09.500748 25957 tablet_copy_client.cc:783] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:09.503340 25957 tablet_copy_client.cc:647] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:09.505326 25957 tablet_copy_client.cc:523] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:09.508744 25957 tablet_bootstrap.cc:492] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:09.506855 25941 tablet_copy_client.cc:783] T db96ccbe81564bb589b0de65977b3321 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:09.509227 25941 tablet_copy_client.cc:647] T db96ccbe81564bb589b0de65977b3321 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:09.511588 25941 tablet_copy_client.cc:523] T db96ccbe81564bb589b0de65977b3321 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:09.513038 25976 ts_tablet_manager.cc:902] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412: Initiating tablet copy from peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:09.511467 25931 tablet_copy_client.cc:783] T d52f4b5c815f4188a09eb12ec5375361 P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:09.517333 25931 tablet_copy_client.cc:647] T d52f4b5c815f4188a09eb12ec5375361 P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:09.510521 25936 tablet_bootstrap.cc:492] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:09.512228 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05bb6b1075f442aaa3bbe005c98ce610" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:09.520049 25001 raft_consensus.cc:2454] T 05bb6b1075f442aaa3bbe005c98ce610 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:54:09.520499 25936 ts_tablet_manager.cc:1367] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.001s
I20240502 01:54:09.521807 25930 tablet_bootstrap.cc:492] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:09.522435 25941 tablet_bootstrap.cc:492] T db96ccbe81564bb589b0de65977b3321 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:09.522733 25930 ts_tablet_manager.cc:1367] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.005s
I20240502 01:54:09.523641 25936 raft_consensus.cc:348] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.524672 22187 leader_election.cc:302] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, c2a28baa7ee2469294d94b7fe5455cff; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:09.525413 25286 raft_consensus.cc:2781] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:09.525573 25286 raft_consensus.cc:482] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:09.525676 25286 raft_consensus.cc:3037] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.526969 25931 tablet_copy_client.cc:523] T d52f4b5c815f4188a09eb12ec5375361 P 28157bc3e678428fb5f93c077d528513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:09.530618 25286 raft_consensus.cc:504] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:09.532127 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05bb6b1075f442aaa3bbe005c98ce610" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:09.532397 24867 raft_consensus.cc:3037] T 05bb6b1075f442aaa3bbe005c98ce610 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.534044 25931 tablet_bootstrap.cc:492] T d52f4b5c815f4188a09eb12ec5375361 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:09.536002 24867 raft_consensus.cc:2454] T 05bb6b1075f442aaa3bbe005c98ce610 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:54:09.536845 22185 leader_election.cc:302] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:09.537514 25621 raft_consensus.cc:2781] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:09.539816 25286 leader_election.cc:288] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:09.544042 25621 raft_consensus.cc:686] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:54:09.544660 25621 consensus_queue.cc:237] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:09.548769 25936 raft_consensus.cc:374] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.549017 25936 raft_consensus.cc:729] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.549448 25936 consensus_queue.cc:260] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.547343 25930 raft_consensus.cc:348] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:09.550043 25930 raft_consensus.cc:374] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.550302 25930 raft_consensus.cc:729] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.550336 21709 catalog_manager.cc:5484] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:09.551088 25936 ts_tablet_manager.cc:1398] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.029s user 0.001s sys 0.002s
I20240502 01:54:09.551666 25930 consensus_queue.cc:260] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:09.530076 25979 ts_tablet_manager.cc:902] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b: Initiating tablet copy from peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:09.548471 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05bb6b1075f442aaa3bbe005c98ce610" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:09.553980 25001 raft_consensus.cc:3037] T 05bb6b1075f442aaa3bbe005c98ce610 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.558190 25930 ts_tablet_manager.cc:1398] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.035s user 0.001s sys 0.001s
I20240502 01:54:09.558336 25001 raft_consensus.cc:2454] T 05bb6b1075f442aaa3bbe005c98ce610 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:54:09.559900 25930 tablet_bootstrap.cc:492] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:09.563727 25930 tablet_bootstrap.cc:654] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.581904 25930 tablet_bootstrap.cc:492] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:09.582180 25930 ts_tablet_manager.cc:1367] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.004s
I20240502 01:54:09.583189 25930 raft_consensus.cc:348] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.583474 25930 raft_consensus.cc:374] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.583612 25930 raft_consensus.cc:729] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.583940 25930 consensus_queue.cc:260] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.587066 25930 ts_tablet_manager.cc:1398] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:54:09.598414 25976 tablet_copy_client.cc:308] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.200:37097
I20240502 01:54:09.599968 25021 tablet_copy_service.cc:140] P 41c70f373c9d49a49891b03a12c6cb2b: Received BeginTabletCopySession request for tablet 992544bbea6547de96298fd32d74b232 from peer 06543b05a16f42099bc86db2a0284412 ({username='slave'} at 127.14.212.194:34591)
I20240502 01:54:09.600219 25021 tablet_copy_service.cc:161] P 41c70f373c9d49a49891b03a12c6cb2b: Beginning new tablet copy session on tablet 992544bbea6547de96298fd32d74b232 from peer 06543b05a16f42099bc86db2a0284412 at {username='slave'} at 127.14.212.194:34591: session id = 06543b05a16f42099bc86db2a0284412-992544bbea6547de96298fd32d74b232
I20240502 01:54:09.605090 25021 tablet_copy_source_session.cc:205] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:09.607743 25976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 992544bbea6547de96298fd32d74b232. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.620020 25976 tablet_copy_client.cc:783] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:09.620607 25976 tablet_copy_client.cc:647] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:09.622037 25979 tablet_copy_client.cc:308] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.195:38879
I20240502 01:54:09.623018 22140 tablet_copy_service.cc:140] P c64840657cdf4b93980bd34120c94d9a: Received BeginTabletCopySession request for tablet daaadb773202454894890c3379c4a58d from peer 41c70f373c9d49a49891b03a12c6cb2b ({username='slave'} at 127.14.212.200:53727)
I20240502 01:54:09.623203 22140 tablet_copy_service.cc:161] P c64840657cdf4b93980bd34120c94d9a: Beginning new tablet copy session on tablet daaadb773202454894890c3379c4a58d from peer 41c70f373c9d49a49891b03a12c6cb2b at {username='slave'} at 127.14.212.200:53727: session id = 41c70f373c9d49a49891b03a12c6cb2b-daaadb773202454894890c3379c4a58d
I20240502 01:54:09.624435 25976 tablet_copy_client.cc:523] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:09.629776 25976 tablet_bootstrap.cc:492] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:09.632508 25941 tablet_bootstrap.cc:492] T db96ccbe81564bb589b0de65977b3321 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:09.633066 25941 tablet_bootstrap.cc:492] T db96ccbe81564bb589b0de65977b3321 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap complete.
I20240502 01:54:09.633421 25941 ts_tablet_manager.cc:1367] T db96ccbe81564bb589b0de65977b3321 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.111s user 0.050s sys 0.006s
I20240502 01:54:09.634486 25941 raft_consensus.cc:348] T db96ccbe81564bb589b0de65977b3321 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.634908 25941 raft_consensus.cc:729] T db96ccbe81564bb589b0de65977b3321 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.635294 25941 consensus_queue.cc:260] T db96ccbe81564bb589b0de65977b3321 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.635504 22140 tablet_copy_source_session.cc:205] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:09.637022 25979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daaadb773202454894890c3379c4a58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.638542 25738 raft_consensus.cc:482] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:09.638679 25738 raft_consensus.cc:504] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.639389 25738 leader_election.cc:288] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:09.640336 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9850769ab02d497891665afb11679201" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:09.640551 24735 raft_consensus.cc:2454] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:09.640909 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9850769ab02d497891665afb11679201" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:09.641152 25001 raft_consensus.cc:2454] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:09.641592 24804 leader_election.cc:302] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:09.643400 25738 raft_consensus.cc:2781] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:09.643527 25738 raft_consensus.cc:482] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:09.643622 25738 raft_consensus.cc:3037] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.646330 25738 raft_consensus.cc:504] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.647740 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9850769ab02d497891665afb11679201" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:09.647964 24735 raft_consensus.cc:3037] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.650100 25979 tablet_copy_client.cc:783] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:09.650305 25409 raft_consensus.cc:482] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:09.650491 25409 raft_consensus.cc:504] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:09.650625 25979 tablet_copy_client.cc:647] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:09.651337 25409 leader_election.cc:288] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:09.651832 24735 raft_consensus.cc:2454] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:09.652222 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab9f7fe67aa41b5971d5df9e4b5d788" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:09.652364 24802 leader_election.cc:302] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:09.652483 24601 raft_consensus.cc:2454] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:54:09.652540 24734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab9f7fe67aa41b5971d5df9e4b5d788" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:09.652693 25582 raft_consensus.cc:2781] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:09.652769 24734 raft_consensus.cc:2454] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:54:09.652859 25582 raft_consensus.cc:686] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:09.653160 25582 consensus_queue.cc:237] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:09.653334 21791 leader_election.cc:302] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:09.653820 25409 raft_consensus.cc:2781] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:09.653944 25409 raft_consensus.cc:482] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:09.654037 25409 raft_consensus.cc:3037] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.654150 25979 tablet_copy_client.cc:523] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:09.654398 25738 leader_election.cc:288] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:09.654920 25941 ts_tablet_manager.cc:1398] T db96ccbe81564bb589b0de65977b3321 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240502 01:54:09.655007 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9850769ab02d497891665afb11679201" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:09.655236 25001 raft_consensus.cc:3037] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.656002 22010 tablet_copy_service.cc:342] P 06543b05a16f42099bc86db2a0284412: Request end of tablet copy session d7449bb1df214bf4a5f93e4631312f08-db96ccbe81564bb589b0de65977b3321 received from {username='slave'} at 127.14.212.199:39887
I20240502 01:54:09.656190 22010 tablet_copy_service.cc:433] P 06543b05a16f42099bc86db2a0284412: ending tablet copy session d7449bb1df214bf4a5f93e4631312f08-db96ccbe81564bb589b0de65977b3321 on tablet db96ccbe81564bb589b0de65977b3321 with peer d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:09.659083 25409 raft_consensus.cc:504] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:09.659636 25001 raft_consensus.cc:2454] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:09.659790 25409 leader_election.cc:288] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:09.661365 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab9f7fe67aa41b5971d5df9e4b5d788" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:09.661639 24735 raft_consensus.cc:3037] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.662657 25976 tablet_bootstrap.cc:492] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:09.663216 25976 tablet_bootstrap.cc:492] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412: Bootstrap complete.
I20240502 01:54:09.663492 25976 ts_tablet_manager.cc:1367] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.008s
I20240502 01:54:09.664461 24735 raft_consensus.cc:2454] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:54:09.664347 25976 raft_consensus.cc:348] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:09.664583 25976 raft_consensus.cc:729] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.664908 25976 consensus_queue.cc:260] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:09.666018 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab9f7fe67aa41b5971d5df9e4b5d788" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:09.666249 24601 raft_consensus.cc:3037] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.666949 25976 ts_tablet_manager.cc:1398] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:09.667876 25021 tablet_copy_service.cc:342] P 41c70f373c9d49a49891b03a12c6cb2b: Request end of tablet copy session 06543b05a16f42099bc86db2a0284412-992544bbea6547de96298fd32d74b232 received from {username='slave'} at 127.14.212.194:34591
I20240502 01:54:09.668072 25021 tablet_copy_service.cc:433] P 41c70f373c9d49a49891b03a12c6cb2b: ending tablet copy session 06543b05a16f42099bc86db2a0284412-992544bbea6547de96298fd32d74b232 on tablet 992544bbea6547de96298fd32d74b232 with peer 06543b05a16f42099bc86db2a0284412
I20240502 01:54:09.668923 24601 raft_consensus.cc:2454] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:54:09.669458 21791 leader_election.cc:302] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:09.675853 25979 tablet_bootstrap.cc:492] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:09.677456 21709 catalog_manager.cc:5484] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:09.678499 25409 raft_consensus.cc:2781] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:09.678735 25409 raft_consensus.cc:686] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:54:09.679092 25409 consensus_queue.cc:237] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:09.699180 21709 catalog_manager.cc:5484] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:09.709204 21854 consensus_queue.cc:237] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
W20240502 01:54:09.722648 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 766241df5ee24264a5cb7176e6c97b91. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:09.724113 21988 raft_consensus.cc:1261] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:09.724658 22117 raft_consensus.cc:1261] T 766241df5ee24264a5cb7176e6c97b91 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:09.725356 25957 tablet_bootstrap.cc:492] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:09.725798 25409 consensus_queue.cc:1035] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:09.725998 25957 tablet_bootstrap.cc:492] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap complete.
I20240502 01:54:09.726541 25957 ts_tablet_manager.cc:1367] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.218s user 0.043s sys 0.013s
I20240502 01:54:09.727491 25409 consensus_queue.cc:1035] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:09.727742 25957 raft_consensus.cc:348] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:09.728060 25957 raft_consensus.cc:729] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.728420 25957 consensus_queue.cc:260] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:09.729127 24601 raft_consensus.cc:1203] T 3cf315f0117444ae9c5390a1e19460d9 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:09.730125 25957 ts_tablet_manager.cc:1398] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:09.730820 22140 tablet_copy_service.cc:342] P c64840657cdf4b93980bd34120c94d9a: Request end of tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-ad3cd6fff977431b9cd5df6d083bef80 received from {username='slave'} at 127.14.212.200:53727
I20240502 01:54:09.730994 22140 tablet_copy_service.cc:433] P c64840657cdf4b93980bd34120c94d9a: ending tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-ad3cd6fff977431b9cd5df6d083bef80 on tablet ad3cd6fff977431b9cd5df6d083bef80 with peer 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:09.736797 25931 tablet_bootstrap.cc:492] T d52f4b5c815f4188a09eb12ec5375361 P 28157bc3e678428fb5f93c077d528513: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:09.740218 25931 tablet_bootstrap.cc:492] T d52f4b5c815f4188a09eb12ec5375361 P 28157bc3e678428fb5f93c077d528513: Bootstrap complete.
I20240502 01:54:09.741353 25931 ts_tablet_manager.cc:1367] T d52f4b5c815f4188a09eb12ec5375361 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.207s user 0.059s sys 0.009s
I20240502 01:54:09.742898 25931 raft_consensus.cc:348] T d52f4b5c815f4188a09eb12ec5375361 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:09.743232 25931 raft_consensus.cc:729] T d52f4b5c815f4188a09eb12ec5375361 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.745018 26006 ts_tablet_manager.cc:902] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a: Initiating tablet copy from peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:09.745004 25931 consensus_queue.cc:260] T d52f4b5c815f4188a09eb12ec5375361 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:09.745819 25931 ts_tablet_manager.cc:1398] T d52f4b5c815f4188a09eb12ec5375361 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:09.746704 21878 tablet_copy_service.cc:342] P 759257ea7bef415ca698e60d82199d77: Request end of tablet copy session 28157bc3e678428fb5f93c077d528513-d52f4b5c815f4188a09eb12ec5375361 received from {username='slave'} at 127.14.212.198:60613
I20240502 01:54:09.746850 21878 tablet_copy_service.cc:433] P 759257ea7bef415ca698e60d82199d77: ending tablet copy session 28157bc3e678428fb5f93c077d528513-d52f4b5c815f4188a09eb12ec5375361 on tablet d52f4b5c815f4188a09eb12ec5375361 with peer 28157bc3e678428fb5f93c077d528513
I20240502 01:54:09.746997 25409 raft_consensus.cc:2932] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:09.747759 26008 ts_tablet_manager.cc:902] T 03fb607ef0d9469daca1d6f3c5259fb2 P 759257ea7bef415ca698e60d82199d77: Initiating tablet copy from peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:09.751638 22118 raft_consensus.cc:2932] T 766241df5ee24264a5cb7176e6c97b91 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:09.757743 21715 catalog_manager.cc:5484] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:09.755836 21988 raft_consensus.cc:2932] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:09.775368 26008 tablet_copy_client.cc:308] T 03fb607ef0d9469daca1d6f3c5259fb2 P 759257ea7bef415ca698e60d82199d77: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.198:42375
I20240502 01:54:09.776554 24755 tablet_copy_service.cc:140] P 28157bc3e678428fb5f93c077d528513: Received BeginTabletCopySession request for tablet 03fb607ef0d9469daca1d6f3c5259fb2 from peer 759257ea7bef415ca698e60d82199d77 ({username='slave'} at 127.14.212.193:45457)
I20240502 01:54:09.776783 24755 tablet_copy_service.cc:161] P 28157bc3e678428fb5f93c077d528513: Beginning new tablet copy session on tablet 03fb607ef0d9469daca1d6f3c5259fb2 from peer 759257ea7bef415ca698e60d82199d77 at {username='slave'} at 127.14.212.193:45457: session id = 759257ea7bef415ca698e60d82199d77-03fb607ef0d9469daca1d6f3c5259fb2
I20240502 01:54:09.780500 24755 tablet_copy_source_session.cc:205] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:09.783258 26008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03fb607ef0d9469daca1d6f3c5259fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.785380 26006 tablet_copy_client.cc:308] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.199:35737
I20240502 01:54:09.797730 25979 tablet_bootstrap.cc:492] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:09.798683 26008 tablet_copy_client.cc:783] T 03fb607ef0d9469daca1d6f3c5259fb2 P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:09.799656 26008 tablet_copy_client.cc:647] T 03fb607ef0d9469daca1d6f3c5259fb2 P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:09.802687 26008 tablet_copy_client.cc:523] T 03fb607ef0d9469daca1d6f3c5259fb2 P 759257ea7bef415ca698e60d82199d77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:09.809073 24867 raft_consensus.cc:1203] T db96ccbe81564bb589b0de65977b3321 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:09.811007 25979 tablet_bootstrap.cc:492] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap complete.
I20240502 01:54:09.811489 25979 ts_tablet_manager.cc:1367] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.136s user 0.044s sys 0.013s
I20240502 01:54:09.812413 26008 tablet_bootstrap.cc:492] T 03fb607ef0d9469daca1d6f3c5259fb2 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:09.812578 21709 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:09.813426 25979 raft_consensus.cc:348] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:09.813689 25979 raft_consensus.cc:729] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.818562 25979 consensus_queue.cc:260] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:09.820622 25979 ts_tablet_manager.cc:1398] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:54:09.823608 22140 tablet_copy_service.cc:342] P c64840657cdf4b93980bd34120c94d9a: Request end of tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-daaadb773202454894890c3379c4a58d received from {username='slave'} at 127.14.212.200:53727
I20240502 01:54:09.823765 22140 tablet_copy_service.cc:433] P c64840657cdf4b93980bd34120c94d9a: ending tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-daaadb773202454894890c3379c4a58d on tablet daaadb773202454894890c3379c4a58d with peer 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:09.839668 25001 raft_consensus.cc:1261] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:09.840481 25738 consensus_queue.cc:1035] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.841989 24735 raft_consensus.cc:1261] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:09.842816 25582 consensus_queue.cc:1035] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.846346 24888 tablet_copy_service.cc:140] P d7449bb1df214bf4a5f93e4631312f08: Received BeginTabletCopySession request for tablet b60b948dae4841dd9c0919f2fcac5299 from peer c64840657cdf4b93980bd34120c94d9a ({username='slave'} at 127.14.212.195:37371)
I20240502 01:54:09.846567 24888 tablet_copy_service.cc:161] P d7449bb1df214bf4a5f93e4631312f08: Beginning new tablet copy session on tablet b60b948dae4841dd9c0919f2fcac5299 from peer c64840657cdf4b93980bd34120c94d9a at {username='slave'} at 127.14.212.195:37371: session id = c64840657cdf4b93980bd34120c94d9a-b60b948dae4841dd9c0919f2fcac5299
I20240502 01:54:09.850023 21709 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 9850769ab02d497891665afb11679201 (table rebalancer_dynamic_table [id=ffb30602844e4d1fb3265fad1e97a6ae]). Expected version 1 got 0
I20240502 01:54:09.850420 24888 tablet_copy_source_session.cc:205] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:09.852777 26006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60b948dae4841dd9c0919f2fcac5299. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.856755 26008 tablet_bootstrap.cc:492] T 03fb607ef0d9469daca1d6f3c5259fb2 P 759257ea7bef415ca698e60d82199d77: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:09.858199 26008 tablet_bootstrap.cc:492] T 03fb607ef0d9469daca1d6f3c5259fb2 P 759257ea7bef415ca698e60d82199d77: Bootstrap complete.
I20240502 01:54:09.858585 26008 ts_tablet_manager.cc:1367] T 03fb607ef0d9469daca1d6f3c5259fb2 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.007s
I20240502 01:54:09.858901 26034 tablet.cc:1722] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:09.860221 26034 tablet.cc:1620] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:09.864044 26006 tablet_copy_client.cc:783] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:09.864423 26006 tablet_copy_client.cc:647] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:09.864617 26008 raft_consensus.cc:348] T 03fb607ef0d9469daca1d6f3c5259fb2 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:09.867645 26006 tablet_copy_client.cc:523] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:09.867507 26008 raft_consensus.cc:729] T 03fb607ef0d9469daca1d6f3c5259fb2 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.868278 26008 consensus_queue.cc:260] T 03fb607ef0d9469daca1d6f3c5259fb2 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:09.874249 25630 consensus_queue.cc:1035] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.877312 24733 raft_consensus.cc:1203] T d52f4b5c815f4188a09eb12ec5375361 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:09.877934 26035 tablet.cc:1722] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:09.878432 26035 tablet.cc:1620] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:09.879335 25000 raft_consensus.cc:1203] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:09.880388 26008 ts_tablet_manager.cc:1398] T 03fb607ef0d9469daca1d6f3c5259fb2 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.017s user 0.002s sys 0.001s
I20240502 01:54:09.882905 25726 consensus_queue.cc:1035] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.894207 21709 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=ffb30602844e4d1fb3265fad1e97a6ae] alter complete (version 1)
I20240502 01:54:09.897578 24755 tablet_copy_service.cc:342] P 28157bc3e678428fb5f93c077d528513: Request end of tablet copy session 759257ea7bef415ca698e60d82199d77-03fb607ef0d9469daca1d6f3c5259fb2 received from {username='slave'} at 127.14.212.193:45457
I20240502 01:54:09.897811 24755 tablet_copy_service.cc:433] P 28157bc3e678428fb5f93c077d528513: ending tablet copy session 759257ea7bef415ca698e60d82199d77-03fb607ef0d9469daca1d6f3c5259fb2 on tablet 03fb607ef0d9469daca1d6f3c5259fb2 with peer 759257ea7bef415ca698e60d82199d77
I20240502 01:54:09.913079 26006 tablet_bootstrap.cc:492] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:09.915241 21709 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 9850769ab02d497891665afb11679201 (table rebalancer_dynamic_table [id=ffb30602844e4d1fb3265fad1e97a6ae]). Expected version 1 got 0
I20240502 01:54:09.920203 21709 catalog_manager.cc:5576] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375) does not have the latest schema for tablet 9850769ab02d497891665afb11679201 (table rebalancer_dynamic_table [id=ffb30602844e4d1fb3265fad1e97a6ae]). Expected version 1 got 0
I20240502 01:54:09.936125 25621 consensus_queue.cc:1035] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.942881 21988 raft_consensus.cc:1203] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:09.943265 21709 catalog_manager.cc:5576] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375) does not have the latest schema for tablet 9850769ab02d497891665afb11679201 (table rebalancer_dynamic_table [id=ffb30602844e4d1fb3265fad1e97a6ae]). Expected version 1 got 0
I20240502 01:54:09.945408 26045 tablet.cc:1722] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:09.946035 26045 tablet.cc:1620] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:09.961241 26006 tablet_bootstrap.cc:492] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:09.961922 26006 tablet_bootstrap.cc:492] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a: Bootstrap complete.
I20240502 01:54:09.962450 26006 ts_tablet_manager.cc:1367] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.002s
I20240502 01:54:09.963959 26006 raft_consensus.cc:348] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:09.964251 26006 raft_consensus.cc:729] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.964560 26006 consensus_queue.cc:260] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:09.966547 26006 ts_tablet_manager.cc:1398] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:09.967397 24888 tablet_copy_service.cc:342] P d7449bb1df214bf4a5f93e4631312f08: Request end of tablet copy session c64840657cdf4b93980bd34120c94d9a-b60b948dae4841dd9c0919f2fcac5299 received from {username='slave'} at 127.14.212.195:37371
I20240502 01:54:09.967602 24888 tablet_copy_service.cc:433] P d7449bb1df214bf4a5f93e4631312f08: ending tablet copy session c64840657cdf4b93980bd34120c94d9a-b60b948dae4841dd9c0919f2fcac5299 on tablet b60b948dae4841dd9c0919f2fcac5299 with peer c64840657cdf4b93980bd34120c94d9a
I20240502 01:54:09.983763 25981 consensus_queue.cc:1035] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.996301 21854 consensus_queue.cc:237] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:10.007968 21709 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:10.008358 25001 raft_consensus.cc:1261] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:10.009058 26002 consensus_queue.cc:1035] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:10.010394 24601 raft_consensus.cc:1261] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:10.012033 26000 consensus_queue.cc:1035] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:54:10.017613 21792 consensus_peers.cc:485] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e00feee2662a48d8a1a57f9e52275154. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:10.019125 26034 tablet.cc:1722] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:10.019670 26034 tablet.cc:1620] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:10.020036 26035 tablet.cc:1722] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:10.020507 26035 tablet.cc:1620] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:10.024832 26045 tablet.cc:1722] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:10.025352 26045 tablet.cc:1620] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:10.026481 26002 raft_consensus.cc:2932] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:10.027230 24601 raft_consensus.cc:2932] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:10.032270 21709 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=ffb30602844e4d1fb3265fad1e97a6ae] alter complete (version 2)
I20240502 01:54:10.034379 21709 catalog_manager.cc:5484] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:10.036641 25001 raft_consensus.cc:2932] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:10.069245 25001 raft_consensus.cc:1203] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:10.078153 21988 consensus_queue.cc:237] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:10.079433 26002 consensus_queue.cc:1035] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:10.080502 26002 consensus_queue.cc:1035] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:10.081717 22249 raft_consensus.cc:1261] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:10.082150 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4b6dfe0e8da4352b136d6f97d473bad. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:10.082530 25791 consensus_queue.cc:1035] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:10.101073 22118 raft_consensus.cc:1261] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:10.101919 26072 consensus_queue.cc:1035] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:10.121115 25791 raft_consensus.cc:2932] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:10.121907 22118 raft_consensus.cc:2932] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:10.129024 21709 catalog_manager.cc:5484] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:10.135634 21709 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:10.138772 22249 raft_consensus.cc:2932] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:10.144861 26034 tablet.cc:1722] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:10.148669 26045 tablet.cc:1722] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:10.149711 26035 tablet.cc:1722] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:10.157380 21715 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=ffb30602844e4d1fb3265fad1e97a6ae] alter complete (version 3)
I20240502 01:54:10.171995 21854 raft_consensus.cc:1203] T 03fb607ef0d9469daca1d6f3c5259fb2 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:10.184367 22117 consensus_queue.cc:237] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:10.187120 21854 raft_consensus.cc:1261] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:10.187189 21988 raft_consensus.cc:1261] T fedbec8860904a2cb5294037b7a4bcad P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:10.187613 22056 consensus_peers.cc:485] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a -> Peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Couldn't send request to peer 41c70f373c9d49a49891b03a12c6cb2b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fedbec8860904a2cb5294037b7a4bcad. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:10.188131 26097 consensus_queue.cc:1035] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:10.188591 25400 consensus_queue.cc:1035] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:10.192255 25400 raft_consensus.cc:2932] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:10.192906 21988 raft_consensus.cc:2932] T fedbec8860904a2cb5294037b7a4bcad P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:10.198109 21854 raft_consensus.cc:2932] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:10.198894 21709 catalog_manager.cc:5484] T fedbec8860904a2cb5294037b7a4bcad P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:10.221786 22117 raft_consensus.cc:1203] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:10.226969 25931 ts_tablet_manager.cc:902] T 766241df5ee24264a5cb7176e6c97b91 P 28157bc3e678428fb5f93c077d528513: Initiating tablet copy from peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:10.240387 21710 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:10.242584 22117 consensus_queue.cc:237] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:10.247578 21988 raft_consensus.cc:1261] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:10.248209 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0eed9ee06a494588a32b4c951d6d3a14. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:10.249089 22249 raft_consensus.cc:1261] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:10.249783 25400 consensus_queue.cc:1035] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:10.250341 25400 consensus_queue.cc:1035] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:10.255501 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 5152c0bcae6946059900d52f0d33ac5b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=9f45904cc4cd41f0962b57ad839bb4c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.256088 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5152c0bcae6946059900d52f0d33ac5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.256412 24979 tablet_service.cc:1467] Processing CreateTablet for tablet cc0f7b64459848019ae2b748dd1ad8dc (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=9f45904cc4cd41f0962b57ad839bb4c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.256852 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0f7b64459848019ae2b748dd1ad8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.261381 24582 tablet_service.cc:1467] Processing CreateTablet for tablet 5152c0bcae6946059900d52f0d33ac5b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=9f45904cc4cd41f0962b57ad839bb4c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.261883 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5152c0bcae6946059900d52f0d33ac5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.264081 21967 tablet_service.cc:1467] Processing CreateTablet for tablet cc0f7b64459848019ae2b748dd1ad8dc (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=9f45904cc4cd41f0962b57ad839bb4c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.264518 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0f7b64459848019ae2b748dd1ad8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.265914 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 5152c0bcae6946059900d52f0d33ac5b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=9f45904cc4cd41f0962b57ad839bb4c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.266391 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5152c0bcae6946059900d52f0d33ac5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.271764 21709 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:10.272019 21709 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:10.279402 21709 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 9850769ab02d497891665afb11679201
I20240502 01:54:10.279865 24713 tablet_service.cc:1467] Processing CreateTablet for tablet cc0f7b64459848019ae2b748dd1ad8dc (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=9f45904cc4cd41f0962b57ad839bb4c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.280393 24713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0f7b64459848019ae2b748dd1ad8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.282063 24981 tablet_service.cc:1514] Processing DeleteTablet for tablet 9850769ab02d497891665afb11679201 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:10 UTC) from {username='slave'} at 127.0.0.1:39964
I20240502 01:54:10.283051 24846 tablet_service.cc:1514] Processing DeleteTablet for tablet 9850769ab02d497891665afb11679201 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:10 UTC) from {username='slave'} at 127.0.0.1:51592
I20240502 01:54:10.285037 24714 tablet_service.cc:1514] Processing DeleteTablet for tablet 9850769ab02d497891665afb11679201 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:10 UTC) from {username='slave'} at 127.0.0.1:41650
I20240502 01:54:10.292176 26122 tablet_bootstrap.cc:492] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:10.295439 26122 tablet_bootstrap.cc:654] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.311712 24712 tablet_service.cc:1467] Processing CreateTablet for tablet 41e68a88b92347e8855b6e5c82e7b7e1 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=9f45904cc4cd41f0962b57ad839bb4c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.313158 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 41e68a88b92347e8855b6e5c82e7b7e1 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=9f45904cc4cd41f0962b57ad839bb4c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.313516 24848 tablet_service.cc:1467] Processing CreateTablet for tablet 41e68a88b92347e8855b6e5c82e7b7e1 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=9f45904cc4cd41f0962b57ad839bb4c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.313750 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41e68a88b92347e8855b6e5c82e7b7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.314074 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41e68a88b92347e8855b6e5c82e7b7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.330343 26124 tablet_bootstrap.cc:492] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:10.333217 24712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41e68a88b92347e8855b6e5c82e7b7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.333398 26123 tablet_replica.cc:330] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:54:10.333704 26123 raft_consensus.cc:2229] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:10.333988 26123 raft_consensus.cc:2258] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:10.335289 26123 ts_tablet_manager.cc:1875] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:10.338788 26131 tablet_bootstrap.cc:492] T 41e68a88b92347e8855b6e5c82e7b7e1 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:10.350498 26131 tablet_bootstrap.cc:654] T 41e68a88b92347e8855b6e5c82e7b7e1 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.353142 26124 tablet_bootstrap.cc:654] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.356276 25931 tablet_copy_client.cc:308] T 766241df5ee24264a5cb7176e6c97b91 P 28157bc3e678428fb5f93c077d528513: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.193:33785
I20240502 01:54:10.357318 26126 tablet_bootstrap.cc:492] T 5152c0bcae6946059900d52f0d33ac5b P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:10.357503 21878 tablet_copy_service.cc:140] P 759257ea7bef415ca698e60d82199d77: Received BeginTabletCopySession request for tablet 766241df5ee24264a5cb7176e6c97b91 from peer 28157bc3e678428fb5f93c077d528513 ({username='slave'} at 127.14.212.198:60613)
I20240502 01:54:10.357712 21878 tablet_copy_service.cc:161] P 759257ea7bef415ca698e60d82199d77: Beginning new tablet copy session on tablet 766241df5ee24264a5cb7176e6c97b91 from peer 28157bc3e678428fb5f93c077d528513 at {username='slave'} at 127.14.212.198:60613: session id = 28157bc3e678428fb5f93c077d528513-766241df5ee24264a5cb7176e6c97b91
I20240502 01:54:10.359326 26126 tablet_bootstrap.cc:654] T 5152c0bcae6946059900d52f0d33ac5b P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.363327 25400 raft_consensus.cc:2932] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:10.364637 26123 ts_tablet_manager.cc:1888] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:10.364750 26123 log.cc:1196] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/9850769ab02d497891665afb11679201
I20240502 01:54:10.365216 26123 ts_tablet_manager.cc:1909] T 9850769ab02d497891665afb11679201 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting consensus metadata
I20240502 01:54:10.365149 22250 raft_consensus.cc:2932] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:10.367144 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet 9850769ab02d497891665afb11679201 (table rebalancer_dynamic_table_new_name [id=ffb30602844e4d1fb3265fad1e97a6ae]) successfully deleted
I20240502 01:54:10.368314 21988 raft_consensus.cc:2932] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:10.372254 26129 tablet_bootstrap.cc:492] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:10.374153 26130 tablet_bootstrap.cc:492] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:10.375500 21878 tablet_copy_source_session.cc:205] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:10.376842 26125 tablet_replica.cc:330] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08: stopping tablet replica
I20240502 01:54:10.379453 26124 tablet_bootstrap.cc:492] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:10.379627 26124 ts_tablet_manager.cc:1367] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.049s user 0.000s sys 0.004s
I20240502 01:54:10.380512 26124 raft_consensus.cc:348] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:10.380738 26124 raft_consensus.cc:374] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.380812 26124 raft_consensus.cc:729] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.381065 26124 consensus_queue.cc:260] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:10.381613 26124 ts_tablet_manager.cc:1398] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:10.388309 26126 tablet_bootstrap.cc:492] T 5152c0bcae6946059900d52f0d33ac5b P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:10.388545 26126 ts_tablet_manager.cc:1367] T 5152c0bcae6946059900d52f0d33ac5b P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.008s
I20240502 01:54:10.389601 26126 raft_consensus.cc:348] T 5152c0bcae6946059900d52f0d33ac5b P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:10.389931 26126 raft_consensus.cc:374] T 5152c0bcae6946059900d52f0d33ac5b P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.390092 26126 raft_consensus.cc:729] T 5152c0bcae6946059900d52f0d33ac5b P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.390614 26126 consensus_queue.cc:260] T 5152c0bcae6946059900d52f0d33ac5b P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:10.391840 26126 ts_tablet_manager.cc:1398] T 5152c0bcae6946059900d52f0d33ac5b P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:10.392314 26129 tablet_bootstrap.cc:654] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.392473 21710 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:10.375041 26127 tablet_replica.cc:330] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:54:10.393693 26125 raft_consensus.cc:2229] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:10.394119 21714 catalog_manager.cc:5484] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:10.395485 26126 tablet_bootstrap.cc:492] T 41e68a88b92347e8855b6e5c82e7b7e1 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:10.396844 25931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 766241df5ee24264a5cb7176e6c97b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.397200 26127 raft_consensus.cc:2229] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:10.397807 26127 raft_consensus.cc:2258] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:10.398394 26006 ts_tablet_manager.cc:902] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a: Initiating tablet copy from peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:10.398370 26125 raft_consensus.cc:2258] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:10.399856 26127 ts_tablet_manager.cc:1875] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:10.388896 26130 tablet_bootstrap.cc:654] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.414006 26131 tablet_bootstrap.cc:492] T 41e68a88b92347e8855b6e5c82e7b7e1 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:10.412366 26126 tablet_bootstrap.cc:654] T 41e68a88b92347e8855b6e5c82e7b7e1 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.415000 26122 tablet_bootstrap.cc:492] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:10.415342 26125 ts_tablet_manager.cc:1875] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:10.415884 26131 ts_tablet_manager.cc:1367] T 41e68a88b92347e8855b6e5c82e7b7e1 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.077s user 0.005s sys 0.000s
I20240502 01:54:10.417315 26131 raft_consensus.cc:348] T 41e68a88b92347e8855b6e5c82e7b7e1 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:10.420764 26131 raft_consensus.cc:374] T 41e68a88b92347e8855b6e5c82e7b7e1 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.420851 26122 ts_tablet_manager.cc:1367] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.129s user 0.007s sys 0.032s
I20240502 01:54:10.421243 26131 raft_consensus.cc:729] T 41e68a88b92347e8855b6e5c82e7b7e1 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.422055 26131 consensus_queue.cc:260] T 41e68a88b92347e8855b6e5c82e7b7e1 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:10.425545 26122 raft_consensus.cc:348] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:10.425944 25931 tablet_copy_client.cc:783] T 766241df5ee24264a5cb7176e6c97b91 P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:10.425968 24846 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa5bf56248e42ae8e006644da6b6fda (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ba03fb8cfce44e7b95a6106e2dd9939a]), partition=
I20240502 01:54:10.426218 25931 tablet_copy_client.cc:647] T 766241df5ee24264a5cb7176e6c97b91 P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:10.426352 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa5bf56248e42ae8e006644da6b6fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.426385 26122 raft_consensus.cc:374] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.426501 26122 raft_consensus.cc:729] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.427079 26122 consensus_queue.cc:260] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:10.430158 26126 tablet_bootstrap.cc:492] T 41e68a88b92347e8855b6e5c82e7b7e1 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:10.430378 26126 ts_tablet_manager.cc:1367] T 41e68a88b92347e8855b6e5c82e7b7e1 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.004s
I20240502 01:54:10.430975 26122 ts_tablet_manager.cc:1398] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240502 01:54:10.431334 24979 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa5bf56248e42ae8e006644da6b6fda (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ba03fb8cfce44e7b95a6106e2dd9939a]), partition=
I20240502 01:54:10.431841 26122 tablet_bootstrap.cc:492] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:10.428694 26127 ts_tablet_manager.cc:1888] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:10.432482 26127 log.cc:1196] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/9850769ab02d497891665afb11679201
I20240502 01:54:10.431880 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa5bf56248e42ae8e006644da6b6fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.433084 26127 ts_tablet_manager.cc:1909] T 9850769ab02d497891665afb11679201 P 28157bc3e678428fb5f93c077d528513: Deleting consensus metadata
I20240502 01:54:10.434690 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet 9850769ab02d497891665afb11679201 (table rebalancer_dynamic_table_new_name [id=ffb30602844e4d1fb3265fad1e97a6ae]) successfully deleted
I20240502 01:54:10.431972 26126 raft_consensus.cc:348] T 41e68a88b92347e8855b6e5c82e7b7e1 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:10.435112 26126 raft_consensus.cc:374] T 41e68a88b92347e8855b6e5c82e7b7e1 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.435261 26126 raft_consensus.cc:729] T 41e68a88b92347e8855b6e5c82e7b7e1 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.435678 26126 consensus_queue.cc:260] T 41e68a88b92347e8855b6e5c82e7b7e1 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:10.435896 26122 tablet_bootstrap.cc:654] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.430837 24714 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa5bf56248e42ae8e006644da6b6fda (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ba03fb8cfce44e7b95a6106e2dd9939a]), partition=
I20240502 01:54:10.437585 24714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa5bf56248e42ae8e006644da6b6fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.441154 26126 ts_tablet_manager.cc:1398] T 41e68a88b92347e8855b6e5c82e7b7e1 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240502 01:54:10.443341 26126 tablet_bootstrap.cc:492] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:10.445441 26126 tablet_bootstrap.cc:654] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.445600 26122 tablet_bootstrap.cc:492] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:10.446557 26125 ts_tablet_manager.cc:1888] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:10.447499 26122 ts_tablet_manager.cc:1367] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.004s
I20240502 01:54:10.449195 26125 log.cc:1196] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/wals/9850769ab02d497891665afb11679201
I20240502 01:54:10.452430 26125 ts_tablet_manager.cc:1909] T 9850769ab02d497891665afb11679201 P d7449bb1df214bf4a5f93e4631312f08: Deleting consensus metadata
I20240502 01:54:10.448664 26122 raft_consensus.cc:348] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:10.453442 26122 raft_consensus.cc:374] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.454732 21700 catalog_manager.cc:4830] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): tablet 9850769ab02d497891665afb11679201 (table rebalancer_dynamic_table_new_name [id=ffb30602844e4d1fb3265fad1e97a6ae]) successfully deleted
I20240502 01:54:10.455152 26122 raft_consensus.cc:729] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.456149 26122 consensus_queue.cc:260] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:10.459022 26122 ts_tablet_manager.cc:1398] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240502 01:54:10.460863 26006 tablet_copy_client.cc:308] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.193:33785
I20240502 01:54:10.461752 26130 tablet_bootstrap.cc:492] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:10.461988 21878 tablet_copy_service.cc:140] P 759257ea7bef415ca698e60d82199d77: Received BeginTabletCopySession request for tablet e00feee2662a48d8a1a57f9e52275154 from peer c64840657cdf4b93980bd34120c94d9a ({username='slave'} at 127.14.212.195:49597)
I20240502 01:54:10.462179 21878 tablet_copy_service.cc:161] P 759257ea7bef415ca698e60d82199d77: Beginning new tablet copy session on tablet e00feee2662a48d8a1a57f9e52275154 from peer c64840657cdf4b93980bd34120c94d9a at {username='slave'} at 127.14.212.195:49597: session id = c64840657cdf4b93980bd34120c94d9a-e00feee2662a48d8a1a57f9e52275154
I20240502 01:54:10.465327 21878 tablet_copy_source_session.cc:205] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:10.467389 26006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00feee2662a48d8a1a57f9e52275154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.469199 26131 ts_tablet_manager.cc:1398] T 41e68a88b92347e8855b6e5c82e7b7e1 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.050s user 0.002s sys 0.000s
I20240502 01:54:10.470971 26131 tablet_bootstrap.cc:492] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:10.475020 26006 tablet_copy_client.cc:783] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:10.475348 26006 tablet_copy_client.cc:647] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:10.476972 26126 tablet_bootstrap.cc:492] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:10.477434 26126 ts_tablet_manager.cc:1367] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.008s
I20240502 01:54:10.477690 26129 tablet_bootstrap.cc:492] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:10.477921 26129 ts_tablet_manager.cc:1367] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.106s user 0.007s sys 0.000s
I20240502 01:54:10.478552 26126 raft_consensus.cc:348] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:10.478796 26006 tablet_copy_client.cc:523] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:10.479027 26129 raft_consensus.cc:348] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:10.479422 26129 raft_consensus.cc:374] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.479575 26129 raft_consensus.cc:729] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.479826 26126 raft_consensus.cc:374] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.480084 26126 raft_consensus.cc:729] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.480578 26126 consensus_queue.cc:260] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:10.482065 26126 ts_tablet_manager.cc:1398] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:10.483103 26126 tablet_bootstrap.cc:492] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:10.484969 26126 tablet_bootstrap.cc:654] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.485496 26006 tablet_bootstrap.cc:492] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:10.492005 26130 ts_tablet_manager.cc:1367] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.118s user 0.007s sys 0.000s
I20240502 01:54:10.493606 26130 raft_consensus.cc:348] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:10.493853 26130 raft_consensus.cc:374] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.493942 26130 raft_consensus.cc:729] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.494235 26130 consensus_queue.cc:260] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:10.495065 26130 ts_tablet_manager.cc:1398] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:10.497442 26131 tablet_bootstrap.cc:654] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.498457 25931 tablet_copy_client.cc:523] T 766241df5ee24264a5cb7176e6c97b91 P 28157bc3e678428fb5f93c077d528513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:10.499104 26126 tablet_bootstrap.cc:492] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:10.499804 26006 tablet_bootstrap.cc:492] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:10.500080 26126 ts_tablet_manager.cc:1367] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.005s
I20240502 01:54:10.500389 26006 tablet_bootstrap.cc:492] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a: Bootstrap complete.
I20240502 01:54:10.500672 26006 ts_tablet_manager.cc:1367] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.008s
I20240502 01:54:10.501366 26126 raft_consensus.cc:348] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:10.501487 26006 raft_consensus.cc:348] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:10.501762 26006 raft_consensus.cc:729] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.502039 26006 consensus_queue.cc:260] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:10.502578 26129 consensus_queue.cc:260] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:10.502795 26006 ts_tablet_manager.cc:1398] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:10.504446 21878 tablet_copy_service.cc:342] P 759257ea7bef415ca698e60d82199d77: Request end of tablet copy session c64840657cdf4b93980bd34120c94d9a-e00feee2662a48d8a1a57f9e52275154 received from {username='slave'} at 127.14.212.195:49597
I20240502 01:54:10.504587 21878 tablet_copy_service.cc:433] P 759257ea7bef415ca698e60d82199d77: ending tablet copy session c64840657cdf4b93980bd34120c94d9a-e00feee2662a48d8a1a57f9e52275154 on tablet e00feee2662a48d8a1a57f9e52275154 with peer c64840657cdf4b93980bd34120c94d9a
I20240502 01:54:10.505138 26126 raft_consensus.cc:374] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.505455 26126 raft_consensus.cc:729] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.506155 26126 consensus_queue.cc:260] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:10.510288 26129 ts_tablet_manager.cc:1398] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20240502 01:54:10.510684 26129 tablet_bootstrap.cc:492] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:10.510751 26126 ts_tablet_manager.cc:1398] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:54:10.517753 26131 tablet_bootstrap.cc:492] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:10.517971 26131 ts_tablet_manager.cc:1367] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.000s
I20240502 01:54:10.521548 26129 tablet_bootstrap.cc:654] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.502357 25931 tablet_bootstrap.cc:492] T 766241df5ee24264a5cb7176e6c97b91 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:10.529366 26131 raft_consensus.cc:348] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:10.530117 26131 raft_consensus.cc:374] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.530254 26131 raft_consensus.cc:729] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.537583 26131 consensus_queue.cc:260] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:10.538797 26131 ts_tablet_manager.cc:1398] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.021s user 0.003s sys 0.000s
I20240502 01:54:10.545897 26129 tablet_bootstrap.cc:492] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:10.546216 26129 ts_tablet_manager.cc:1367] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20240502 01:54:10.547142 26129 raft_consensus.cc:348] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:10.547394 26129 raft_consensus.cc:374] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.547514 26129 raft_consensus.cc:729] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.547783 26129 consensus_queue.cc:260] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:10.548535 26129 ts_tablet_manager.cc:1398] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:10.570991 26041 raft_consensus.cc:482] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:10.571247 26041 raft_consensus.cc:504] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:10.572194 26041 leader_election.cc:288] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:10.573092 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0f7b64459848019ae2b748dd1ad8dc" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:10.573355 25001 raft_consensus.cc:2454] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:10.574618 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0f7b64459848019ae2b748dd1ad8dc" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:54:10.574883 21988 raft_consensus.cc:2454] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:10.575539 24671 leader_election.cc:302] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:10.576253 26041 raft_consensus.cc:2781] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:10.576382 26041 raft_consensus.cc:482] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:10.576471 26041 raft_consensus.cc:3037] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.579815 26041 raft_consensus.cc:504] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:10.581076 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0f7b64459848019ae2b748dd1ad8dc" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:10.581308 25001 raft_consensus.cc:3037] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.584358 25001 raft_consensus.cc:2454] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:10.585798 24671 leader_election.cc:302] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:10.588773 26041 leader_election.cc:288] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:10.589720 26041 raft_consensus.cc:2781] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:10.590348 26041 raft_consensus.cc:686] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:10.590915 26041 consensus_queue.cc:237] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:10.595798 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0f7b64459848019ae2b748dd1ad8dc" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:54:10.596071 21988 raft_consensus.cc:3037] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.599519 26155 ts_tablet_manager.cc:902] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08: Initiating tablet copy from peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:10.603760 21988 raft_consensus.cc:2454] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:10.618939 21715 catalog_manager.cc:5484] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:10.622624 25931 tablet_bootstrap.cc:492] T 766241df5ee24264a5cb7176e6c97b91 P 28157bc3e678428fb5f93c077d528513: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:10.623239 25931 tablet_bootstrap.cc:492] T 766241df5ee24264a5cb7176e6c97b91 P 28157bc3e678428fb5f93c077d528513: Bootstrap complete.
I20240502 01:54:10.623540 25931 ts_tablet_manager.cc:1367] T 766241df5ee24264a5cb7176e6c97b91 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.121s user 0.052s sys 0.007s
I20240502 01:54:10.624647 25931 raft_consensus.cc:348] T 766241df5ee24264a5cb7176e6c97b91 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:10.624984 25931 raft_consensus.cc:729] T 766241df5ee24264a5cb7176e6c97b91 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.625310 25931 consensus_queue.cc:260] T 766241df5ee24264a5cb7176e6c97b91 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:10.626029 25931 ts_tablet_manager.cc:1398] T 766241df5ee24264a5cb7176e6c97b91 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:10.626984 21878 tablet_copy_service.cc:342] P 759257ea7bef415ca698e60d82199d77: Request end of tablet copy session 28157bc3e678428fb5f93c077d528513-766241df5ee24264a5cb7176e6c97b91 received from {username='slave'} at 127.14.212.198:60613
I20240502 01:54:10.627175 21878 tablet_copy_service.cc:433] P 759257ea7bef415ca698e60d82199d77: ending tablet copy session 28157bc3e678428fb5f93c077d528513-766241df5ee24264a5cb7176e6c97b91 on tablet 766241df5ee24264a5cb7176e6c97b91 with peer 28157bc3e678428fb5f93c077d528513
I20240502 01:54:10.636353 26155 tablet_copy_client.cc:308] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:44411
I20240502 01:54:10.637307 22010 tablet_copy_service.cc:140] P 06543b05a16f42099bc86db2a0284412: Received BeginTabletCopySession request for tablet d4b6dfe0e8da4352b136d6f97d473bad from peer d7449bb1df214bf4a5f93e4631312f08 ({username='slave'} at 127.14.212.199:39887)
I20240502 01:54:10.637497 22010 tablet_copy_service.cc:161] P 06543b05a16f42099bc86db2a0284412: Beginning new tablet copy session on tablet d4b6dfe0e8da4352b136d6f97d473bad from peer d7449bb1df214bf4a5f93e4631312f08 at {username='slave'} at 127.14.212.199:39887: session id = d7449bb1df214bf4a5f93e4631312f08-d4b6dfe0e8da4352b136d6f97d473bad
I20240502 01:54:10.640064 22010 tablet_copy_source_session.cc:205] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:10.641892 26155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b6dfe0e8da4352b136d6f97d473bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.651525 26155 tablet_copy_client.cc:783] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:10.651644 26041 raft_consensus.cc:482] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:10.651922 26155 tablet_copy_client.cc:647] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:10.651863 26041 raft_consensus.cc:504] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:10.652561 26041 leader_election.cc:288] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:10.653321 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa5bf56248e42ae8e006644da6b6fda" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:10.653399 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa5bf56248e42ae8e006644da6b6fda" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:10.653537 24868 raft_consensus.cc:2454] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:10.653662 25001 raft_consensus.cc:2454] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:10.654058 24669 leader_election.cc:302] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:10.654397 25621 raft_consensus.cc:482] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:10.654451 26155 tablet_copy_client.cc:523] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:10.654646 25621 raft_consensus.cc:504] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:10.655499 25621 leader_election.cc:288] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:10.657199 26041 raft_consensus.cc:2781] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:10.657318 26041 raft_consensus.cc:482] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:10.657438 26041 raft_consensus.cc:3037] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.658059 26155 tablet_bootstrap.cc:492] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:10.660019 26041 raft_consensus.cc:504] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:10.660243 21988 consensus_queue.cc:237] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:10.661489 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa5bf56248e42ae8e006644da6b6fda" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:10.661727 24868 raft_consensus.cc:3037] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.665555 21854 raft_consensus.cc:1261] T ec67f45b2b604c129d8572c5ca6acbc4 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:10.667524 26079 consensus_queue.cc:1035] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:10.667690 22117 raft_consensus.cc:1261] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:10.670706 26079 consensus_queue.cc:1035] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:10.671999 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e68a88b92347e8855b6e5c82e7b7e1" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:10.672284 24867 raft_consensus.cc:2454] T 41e68a88b92347e8855b6e5c82e7b7e1 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:54:10.675371 22185 leader_election.cc:302] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:10.676344 25981 raft_consensus.cc:2781] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:10.676512 25981 raft_consensus.cc:482] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:10.676622 25981 raft_consensus.cc:3037] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.679119 25981 raft_consensus.cc:504] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:10.679693 25981 leader_election.cc:288] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:10.680222 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e68a88b92347e8855b6e5c82e7b7e1" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:10.680475 24867 raft_consensus.cc:3037] T 41e68a88b92347e8855b6e5c82e7b7e1 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.684649 24867 raft_consensus.cc:2454] T 41e68a88b92347e8855b6e5c82e7b7e1 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:54:10.685226 22185 leader_election.cc:302] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:10.686815 25981 raft_consensus.cc:2781] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:10.687122 25981 raft_consensus.cc:686] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:54:10.686584 26079 raft_consensus.cc:2932] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:10.687651 25981 consensus_queue.cc:237] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:10.696484 21854 raft_consensus.cc:2932] T ec67f45b2b604c129d8572c5ca6acbc4 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:10.701918 21715 catalog_manager.cc:5484] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:10.704133 24868 raft_consensus.cc:2454] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:10.704771 24669 leader_election.cc:302] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:10.704444 22118 raft_consensus.cc:2932] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:10.705202 26159 raft_consensus.cc:2781] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:10.707688 21709 catalog_manager.cc:5484] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:10.713354 26041 leader_election.cc:288] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:10.725414 26159 raft_consensus.cc:686] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:10.725986 26159 consensus_queue.cc:237] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:10.734627 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa5bf56248e42ae8e006644da6b6fda" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:10.734993 25000 raft_consensus.cc:3037] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.738569 25000 raft_consensus.cc:2454] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:10.739956 21715 catalog_manager.cc:5484] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:10.745401 24735 raft_consensus.cc:1203] T 766241df5ee24264a5cb7176e6c97b91 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:10.754743 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e68a88b92347e8855b6e5c82e7b7e1" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:10.755095 24735 raft_consensus.cc:2454] T 41e68a88b92347e8855b6e5c82e7b7e1 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:54:10.755913 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e68a88b92347e8855b6e5c82e7b7e1" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:10.756134 24735 raft_consensus.cc:3037] T 41e68a88b92347e8855b6e5c82e7b7e1 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.760031 24735 raft_consensus.cc:2454] T 41e68a88b92347e8855b6e5c82e7b7e1 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
W20240502 01:54:10.765515 21923 consensus_peers.cc:485] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 -> Peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Couldn't send request to peer 0c64d40b5d7644bd8b29feaa80eab72f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec67f45b2b604c129d8572c5ca6acbc4. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:10.768369 26184 ts_tablet_manager.cc:902] T fedbec8860904a2cb5294037b7a4bcad P 41c70f373c9d49a49891b03a12c6cb2b: Initiating tablet copy from peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:10.792330 26184 tablet_copy_client.cc:308] T fedbec8860904a2cb5294037b7a4bcad P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.195:38879
I20240502 01:54:10.793270 22140 tablet_copy_service.cc:140] P c64840657cdf4b93980bd34120c94d9a: Received BeginTabletCopySession request for tablet fedbec8860904a2cb5294037b7a4bcad from peer 41c70f373c9d49a49891b03a12c6cb2b ({username='slave'} at 127.14.212.200:53727)
I20240502 01:54:10.793498 22140 tablet_copy_service.cc:161] P c64840657cdf4b93980bd34120c94d9a: Beginning new tablet copy session on tablet fedbec8860904a2cb5294037b7a4bcad from peer 41c70f373c9d49a49891b03a12c6cb2b at {username='slave'} at 127.14.212.200:53727: session id = 41c70f373c9d49a49891b03a12c6cb2b-fedbec8860904a2cb5294037b7a4bcad
I20240502 01:54:10.795571 26155 tablet_bootstrap.cc:492] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:10.796037 26155 tablet_bootstrap.cc:492] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08: Bootstrap complete.
I20240502 01:54:10.796453 26155 ts_tablet_manager.cc:1367] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.138s user 0.049s sys 0.050s
I20240502 01:54:10.796566 25763 raft_consensus.cc:482] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:10.796857 22140 tablet_copy_source_session.cc:205] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:10.796778 25763 raft_consensus.cc:504] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:10.797425 26155 raft_consensus.cc:348] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:10.797545 25763 leader_election.cc:288] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:10.797679 26155 raft_consensus.cc:729] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.797971 26155 consensus_queue.cc:260] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:10.799017 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5152c0bcae6946059900d52f0d33ac5b" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:10.799152 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5152c0bcae6946059900d52f0d33ac5b" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:10.799289 24735 raft_consensus.cc:2454] T 5152c0bcae6946059900d52f0d33ac5b P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:10.799401 22250 raft_consensus.cc:2454] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:10.799551 26155 ts_tablet_manager.cc:1398] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:10.799938 26184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fedbec8860904a2cb5294037b7a4bcad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.800025 24537 leader_election.cc:302] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:54:10.800218 22010 tablet_copy_service.cc:342] P 06543b05a16f42099bc86db2a0284412: Request end of tablet copy session d7449bb1df214bf4a5f93e4631312f08-d4b6dfe0e8da4352b136d6f97d473bad received from {username='slave'} at 127.14.212.199:39887
I20240502 01:54:10.800396 22010 tablet_copy_service.cc:433] P 06543b05a16f42099bc86db2a0284412: ending tablet copy session d7449bb1df214bf4a5f93e4631312f08-d4b6dfe0e8da4352b136d6f97d473bad on tablet d4b6dfe0e8da4352b136d6f97d473bad with peer d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:10.801512 25763 raft_consensus.cc:2781] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:10.801645 25763 raft_consensus.cc:482] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:10.801746 25763 raft_consensus.cc:3037] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.804665 25763 raft_consensus.cc:504] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:10.806931 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5152c0bcae6946059900d52f0d33ac5b" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:10.807190 24735 raft_consensus.cc:3037] T 5152c0bcae6946059900d52f0d33ac5b P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.807226 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5152c0bcae6946059900d52f0d33ac5b" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:10.807463 22250 raft_consensus.cc:3037] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.807754 26184 tablet_copy_client.cc:783] T fedbec8860904a2cb5294037b7a4bcad P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:10.808096 26184 tablet_copy_client.cc:647] T fedbec8860904a2cb5294037b7a4bcad P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:10.810024 24735 raft_consensus.cc:2454] T 5152c0bcae6946059900d52f0d33ac5b P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:10.810456 24537 leader_election.cc:302] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:54:10.810250 22250 raft_consensus.cc:2454] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:10.811203 25763 leader_election.cc:288] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:10.811231 26197 raft_consensus.cc:2781] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:10.812350 26197 raft_consensus.cc:686] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:54:10.812945 26197 consensus_queue.cc:237] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:10.814064 26184 tablet_copy_client.cc:523] T fedbec8860904a2cb5294037b7a4bcad P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:10.818445 26184 tablet_bootstrap.cc:492] T fedbec8860904a2cb5294037b7a4bcad P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:10.825815 21715 catalog_manager.cc:5484] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:10.842079 25931 ts_tablet_manager.cc:902] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513: Initiating tablet copy from peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:10.883159 25931 tablet_copy_client.cc:308] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.195:38879
I20240502 01:54:10.884058 22140 tablet_copy_service.cc:140] P c64840657cdf4b93980bd34120c94d9a: Received BeginTabletCopySession request for tablet 0eed9ee06a494588a32b4c951d6d3a14 from peer 28157bc3e678428fb5f93c077d528513 ({username='slave'} at 127.14.212.198:48493)
I20240502 01:54:10.884265 22140 tablet_copy_service.cc:161] P c64840657cdf4b93980bd34120c94d9a: Beginning new tablet copy session on tablet 0eed9ee06a494588a32b4c951d6d3a14 from peer 28157bc3e678428fb5f93c077d528513 at {username='slave'} at 127.14.212.198:48493: session id = 28157bc3e678428fb5f93c077d528513-0eed9ee06a494588a32b4c951d6d3a14
I20240502 01:54:10.888245 22140 tablet_copy_source_session.cc:205] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:10.889782 25931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eed9ee06a494588a32b4c951d6d3a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.900976 25931 tablet_copy_client.cc:783] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:10.901386 25931 tablet_copy_client.cc:647] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:10.903388 25931 tablet_copy_client.cc:523] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:10.909297 25931 tablet_bootstrap.cc:492] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:10.914448 21715 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:10.948014 25001 raft_consensus.cc:1261] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:10.949110 24868 raft_consensus.cc:1261] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:10.949767 26159 consensus_queue.cc:1035] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:10.950845 26159 consensus_queue.cc:1035] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:10.967828 21715 catalog_manager.cc:5576] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737) does not have the latest schema for tablet 7aa5bf56248e42ae8e006644da6b6fda (table rebalancer_dynamic_table [id=ba03fb8cfce44e7b95a6106e2dd9939a]). Expected version 1 got 0
I20240502 01:54:10.967881 24867 raft_consensus.cc:1203] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:10.970842 26220 tablet.cc:1722] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:10.971488 26220 tablet.cc:1620] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:10.973291 26183 consensus_queue.cc:1035] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:10.983137 21715 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=ba03fb8cfce44e7b95a6106e2dd9939a] alter complete (version 1)
I20240502 01:54:10.984613 26183 consensus_queue.cc:1035] T cc0f7b64459848019ae2b748dd1ad8dc P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:10.993932 22118 raft_consensus.cc:1203] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:11.002367 26219 tablet.cc:1722] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:11.009470 26219 tablet.cc:1620] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:11.012900 26184 tablet_bootstrap.cc:492] T fedbec8860904a2cb5294037b7a4bcad P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:11.023977 25931 tablet_bootstrap.cc:492] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:11.024542 25931 tablet_bootstrap.cc:492] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513: Bootstrap complete.
I20240502 01:54:11.024911 25931 ts_tablet_manager.cc:1367] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.116s user 0.045s sys 0.008s
I20240502 01:54:11.027189 25931 raft_consensus.cc:348] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:11.027474 25931 raft_consensus.cc:729] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.030810 25931 consensus_queue.cc:260] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:11.032502 26184 tablet_bootstrap.cc:492] T fedbec8860904a2cb5294037b7a4bcad P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap complete.
I20240502 01:54:11.033161 26184 ts_tablet_manager.cc:1367] T fedbec8860904a2cb5294037b7a4bcad P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.215s user 0.064s sys 0.011s
I20240502 01:54:11.043941 21714 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 7aa5bf56248e42ae8e006644da6b6fda (table rebalancer_dynamic_table [id=ba03fb8cfce44e7b95a6106e2dd9939a]). Expected version 1 got 0
I20240502 01:54:11.044668 25931 ts_tablet_manager.cc:1398] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240502 01:54:11.046639 22140 tablet_copy_service.cc:342] P c64840657cdf4b93980bd34120c94d9a: Request end of tablet copy session 28157bc3e678428fb5f93c077d528513-0eed9ee06a494588a32b4c951d6d3a14 received from {username='slave'} at 127.14.212.198:48493
I20240502 01:54:11.046865 22140 tablet_copy_service.cc:433] P c64840657cdf4b93980bd34120c94d9a: ending tablet copy session 28157bc3e678428fb5f93c077d528513-0eed9ee06a494588a32b4c951d6d3a14 on tablet 0eed9ee06a494588a32b4c951d6d3a14 with peer 28157bc3e678428fb5f93c077d528513
I20240502 01:54:11.049711 26227 tablet.cc:1722] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:11.051278 26227 tablet.cc:1620] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:11.037421 26184 raft_consensus.cc:348] T fedbec8860904a2cb5294037b7a4bcad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:11.054270 26184 raft_consensus.cc:729] T fedbec8860904a2cb5294037b7a4bcad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.056834 26184 consensus_queue.cc:260] T fedbec8860904a2cb5294037b7a4bcad P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:11.057653 26184 ts_tablet_manager.cc:1398] T fedbec8860904a2cb5294037b7a4bcad P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20240502 01:54:11.058611 22140 tablet_copy_service.cc:342] P c64840657cdf4b93980bd34120c94d9a: Request end of tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-fedbec8860904a2cb5294037b7a4bcad received from {username='slave'} at 127.14.212.200:53727
I20240502 01:54:11.058759 22140 tablet_copy_service.cc:433] P c64840657cdf4b93980bd34120c94d9a: ending tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-fedbec8860904a2cb5294037b7a4bcad on tablet fedbec8860904a2cb5294037b7a4bcad with peer 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:11.063865 26228 ts_tablet_manager.cc:902] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f: Initiating tablet copy from peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:11.064097 21988 consensus_queue.cc:237] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.067212 26228 tablet_copy_client.cc:308] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:44411
I20240502 01:54:11.069823 21854 raft_consensus.cc:1261] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:11.070428 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f8200b044ef44e1a0451baf2b1485d2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:11.070573 26182 consensus_queue.cc:1035] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:11.072299 22250 raft_consensus.cc:1261] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:11.074755 26182 consensus_queue.cc:1035] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:11.077299 22010 tablet_copy_service.cc:140] P 06543b05a16f42099bc86db2a0284412: Received BeginTabletCopySession request for tablet ec67f45b2b604c129d8572c5ca6acbc4 from peer 0c64d40b5d7644bd8b29feaa80eab72f ({username='slave'} at 127.14.212.197:35233)
I20240502 01:54:11.077451 22010 tablet_copy_service.cc:161] P 06543b05a16f42099bc86db2a0284412: Beginning new tablet copy session on tablet ec67f45b2b604c129d8572c5ca6acbc4 from peer 0c64d40b5d7644bd8b29feaa80eab72f at {username='slave'} at 127.14.212.197:35233: session id = 0c64d40b5d7644bd8b29feaa80eab72f-ec67f45b2b604c129d8572c5ca6acbc4
I20240502 01:54:11.079739 22010 tablet_copy_source_session.cc:205] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:11.081049 26228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec67f45b2b604c129d8572c5ca6acbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.080677 26175 raft_consensus.cc:2932] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:11.081422 22249 raft_consensus.cc:2932] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:11.086624 21714 catalog_manager.cc:5484] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:11.090848 21854 raft_consensus.cc:2932] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:11.094144 26228 tablet_copy_client.cc:783] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:11.094475 26228 tablet_copy_client.cc:647] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:11.095137 21715 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:11.100957 26228 tablet_copy_client.cc:523] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:11.104262 26219 tablet.cc:1722] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:11.104789 26219 tablet.cc:1620] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:11.106281 26227 tablet.cc:1722] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:11.106920 26227 tablet.cc:1620] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:11.106994 26220 tablet.cc:1722] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:11.107474 26220 tablet.cc:1620] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:11.115767 26228 tablet_bootstrap.cc:492] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:11.119805 21715 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=ba03fb8cfce44e7b95a6106e2dd9939a] alter complete (version 2)
I20240502 01:54:11.144788 25981 consensus_queue.cc:1035] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:11.165915 25981 consensus_queue.cc:1035] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:11.175760 26228 tablet_bootstrap.cc:492] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:11.176221 26228 tablet_bootstrap.cc:492] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap complete.
I20240502 01:54:11.176489 26228 ts_tablet_manager.cc:1367] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.061s user 0.044s sys 0.008s
I20240502 01:54:11.177459 26228 raft_consensus.cc:348] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:11.177852 26228 raft_consensus.cc:729] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.178197 26228 consensus_queue.cc:260] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:11.178833 26228 ts_tablet_manager.cc:1398] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:54:11.179680 22010 tablet_copy_service.cc:342] P 06543b05a16f42099bc86db2a0284412: Request end of tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-ec67f45b2b604c129d8572c5ca6acbc4 received from {username='slave'} at 127.14.212.197:35233
I20240502 01:54:11.179841 22010 tablet_copy_service.cc:433] P 06543b05a16f42099bc86db2a0284412: ending tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-ec67f45b2b604c129d8572c5ca6acbc4 on tablet ec67f45b2b604c129d8572c5ca6acbc4 with peer 0c64d40b5d7644bd8b29feaa80eab72f
I20240502 01:54:11.187011 22118 consensus_queue.cc:237] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:11.189448 21854 raft_consensus.cc:1261] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:11.189632 22249 raft_consensus.cc:1261] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:11.190088 25400 consensus_queue.cc:1035] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:11.190418 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Couldn't send request to peer 0c64d40b5d7644bd8b29feaa80eab72f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bceab60366b4ade98f86072d391e7f4. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:11.190502 26135 consensus_queue.cc:1035] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:11.195802 26135 raft_consensus.cc:2932] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:11.196395 21857 raft_consensus.cc:2932] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:11.205293 21709 catalog_manager.cc:5484] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:11.207592 22249 raft_consensus.cc:2932] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:11.209839 25000 raft_consensus.cc:1203] T fedbec8860904a2cb5294037b7a4bcad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:11.225145 21709 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:11.236948 26219 tablet.cc:1722] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:11.239223 26227 tablet.cc:1722] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:11.246592 21709 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=ba03fb8cfce44e7b95a6106e2dd9939a] alter complete (version 3)
I20240502 01:54:11.247009 26220 tablet.cc:1722] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:11.330144 26197 consensus_queue.cc:1035] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:11.347901 26283 consensus_queue.cc:1035] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 01:54:11.354663 21710 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:11.354977 21710 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:11.360641 21710 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 7aa5bf56248e42ae8e006644da6b6fda
I20240502 01:54:11.361817 24979 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa5bf56248e42ae8e006644da6b6fda with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:11 UTC) from {username='slave'} at 127.0.0.1:39964
I20240502 01:54:11.362430 24714 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa5bf56248e42ae8e006644da6b6fda with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:11 UTC) from {username='slave'} at 127.0.0.1:41650
I20240502 01:54:11.362897 24846 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa5bf56248e42ae8e006644da6b6fda with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:11 UTC) from {username='slave'} at 127.0.0.1:51592
I20240502 01:54:11.374864 26297 tablet_replica.cc:330] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:54:11.375749 26296 tablet_replica.cc:330] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:54:11.375298 26297 raft_consensus.cc:2229] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:11.376737 26296 raft_consensus.cc:2229] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:11.376396 26297 raft_consensus.cc:2258] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:11.377298 26296 raft_consensus.cc:2258] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:11.378350 26297 ts_tablet_manager.cc:1875] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:11.379747 26296 ts_tablet_manager.cc:1875] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:11.380479 24735 raft_consensus.cc:1203] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:11.389335 26298 tablet_replica.cc:330] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08: stopping tablet replica
I20240502 01:54:11.389758 26298 raft_consensus.cc:2229] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:11.390096 26298 raft_consensus.cc:2258] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:11.391135 26298 ts_tablet_manager.cc:1875] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:11.402330 26298 ts_tablet_manager.cc:1888] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:11.402576 26298 log.cc:1196] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/wals/7aa5bf56248e42ae8e006644da6b6fda
I20240502 01:54:11.403170 26298 ts_tablet_manager.cc:1909] T 7aa5bf56248e42ae8e006644da6b6fda P d7449bb1df214bf4a5f93e4631312f08: Deleting consensus metadata
I20240502 01:54:11.403539 26297 ts_tablet_manager.cc:1888] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:11.403786 26297 log.cc:1196] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/7aa5bf56248e42ae8e006644da6b6fda
I20240502 01:54:11.404343 26297 ts_tablet_manager.cc:1909] T 7aa5bf56248e42ae8e006644da6b6fda P 28157bc3e678428fb5f93c077d528513: Deleting consensus metadata
I20240502 01:54:11.404124 26296 ts_tablet_manager.cc:1888] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:11.404518 26296 log.cc:1196] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/7aa5bf56248e42ae8e006644da6b6fda
I20240502 01:54:11.405284 21700 catalog_manager.cc:4830] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): tablet 7aa5bf56248e42ae8e006644da6b6fda (table rebalancer_dynamic_table_new_name [id=ba03fb8cfce44e7b95a6106e2dd9939a]) successfully deleted
I20240502 01:54:11.405408 26296 ts_tablet_manager.cc:1909] T 7aa5bf56248e42ae8e006644da6b6fda P 41c70f373c9d49a49891b03a12c6cb2b: Deleting consensus metadata
I20240502 01:54:11.405622 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet 7aa5bf56248e42ae8e006644da6b6fda (table rebalancer_dynamic_table_new_name [id=ba03fb8cfce44e7b95a6106e2dd9939a]) successfully deleted
I20240502 01:54:11.407194 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet 7aa5bf56248e42ae8e006644da6b6fda (table rebalancer_dynamic_table_new_name [id=ba03fb8cfce44e7b95a6106e2dd9939a]) successfully deleted
I20240502 01:54:11.414572 21710 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:11.425544 24582 tablet_service.cc:1467] Processing CreateTablet for tablet 5a16aff54e6f4975a97776c214890f57 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=382119de47e04360a975948cedb418bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.425921 24846 tablet_service.cc:1467] Processing CreateTablet for tablet 5a16aff54e6f4975a97776c214890f57 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=382119de47e04360a975948cedb418bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.426101 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a16aff54e6f4975a97776c214890f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.426398 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a16aff54e6f4975a97776c214890f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.429940 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 5a16aff54e6f4975a97776c214890f57 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=382119de47e04360a975948cedb418bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.430439 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a16aff54e6f4975a97776c214890f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.437176 24581 tablet_service.cc:1467] Processing CreateTablet for tablet feffa925599b4d6aafa48e09e4f6467a (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=382119de47e04360a975948cedb418bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.437672 24581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feffa925599b4d6aafa48e09e4f6467a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.437685 24979 tablet_service.cc:1467] Processing CreateTablet for tablet feffa925599b4d6aafa48e09e4f6467a (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=382119de47e04360a975948cedb418bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.438205 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feffa925599b4d6aafa48e09e4f6467a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.441651 26301 tablet_bootstrap.cc:492] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:11.443881 26301 tablet_bootstrap.cc:654] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.446239 24848 tablet_service.cc:1467] Processing CreateTablet for tablet feffa925599b4d6aafa48e09e4f6467a (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=382119de47e04360a975948cedb418bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.446419 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 8b377c7402194b42a8eabc747f668fb2 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=382119de47e04360a975948cedb418bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.446846 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feffa925599b4d6aafa48e09e4f6467a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.446987 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b377c7402194b42a8eabc747f668fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.449666 24714 tablet_service.cc:1467] Processing CreateTablet for tablet 8b377c7402194b42a8eabc747f668fb2 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=382119de47e04360a975948cedb418bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.450351 24714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b377c7402194b42a8eabc747f668fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.453533 24847 tablet_service.cc:1467] Processing CreateTablet for tablet 8b377c7402194b42a8eabc747f668fb2 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=382119de47e04360a975948cedb418bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.454164 24847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b377c7402194b42a8eabc747f668fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.465997 21710 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:11.483709 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 149912f54ab74daf97652148e4ede2be (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ca8fb519fdb343f09b50f29586ecfb30]), partition=
I20240502 01:54:11.484339 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 149912f54ab74daf97652148e4ede2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.485402 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 149912f54ab74daf97652148e4ede2be (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ca8fb519fdb343f09b50f29586ecfb30]), partition=
I20240502 01:54:11.485873 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 149912f54ab74daf97652148e4ede2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.489892 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 149912f54ab74daf97652148e4ede2be (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ca8fb519fdb343f09b50f29586ecfb30]), partition=
I20240502 01:54:11.490448 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 149912f54ab74daf97652148e4ede2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.446197 26303 tablet_bootstrap.cc:492] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:11.518704 26308 tablet_bootstrap.cc:492] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:11.526933 26303 tablet_bootstrap.cc:654] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.535928 26305 ts_tablet_manager.cc:902] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08: Initiating tablet copy from peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:11.537060 26307 tablet_bootstrap.cc:492] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:11.539453 26307 tablet_bootstrap.cc:654] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.541065 26310 tablet_bootstrap.cc:492] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:11.542347 26308 tablet_bootstrap.cc:654] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.543113 26310 tablet_bootstrap.cc:654] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.544627 26311 tablet_bootstrap.cc:492] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:11.546785 26311 tablet_bootstrap.cc:654] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.551972 26303 tablet_bootstrap.cc:492] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:11.552259 26303 ts_tablet_manager.cc:1367] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.106s user 0.005s sys 0.000s
I20240502 01:54:11.552502 26311 tablet_bootstrap.cc:492] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:11.552675 26311 ts_tablet_manager.cc:1367] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 01:54:11.553469 26311 raft_consensus.cc:348] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:11.553699 26311 raft_consensus.cc:374] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.553802 26311 raft_consensus.cc:729] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.554765 26303 raft_consensus.cc:348] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.555023 26310 tablet_bootstrap.cc:492] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:11.555346 26303 raft_consensus.cc:374] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.555508 26303 raft_consensus.cc:729] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.556521 26303 consensus_queue.cc:260] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.557219 26302 tablet_bootstrap.cc:492] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:11.558331 26307 tablet_bootstrap.cc:492] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:11.558331 26310 ts_tablet_manager.cc:1367] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240502 01:54:11.558794 26305 tablet_copy_client.cc:308] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:44411
I20240502 01:54:11.562428 26310 raft_consensus.cc:348] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:11.562721 26310 raft_consensus.cc:374] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.562817 26310 raft_consensus.cc:729] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.563138 26307 ts_tablet_manager.cc:1367] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240502 01:54:11.563565 26311 consensus_queue.cc:260] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:11.563962 26307 raft_consensus.cc:348] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.564179 26307 raft_consensus.cc:374] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.564270 26307 raft_consensus.cc:729] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.564496 26307 consensus_queue.cc:260] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.564683 26302 tablet_bootstrap.cc:654] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.565150 26307 ts_tablet_manager.cc:1398] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:11.567917 26311 ts_tablet_manager.cc:1398] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.015s user 0.000s sys 0.003s
I20240502 01:54:11.567205 26310 consensus_queue.cc:260] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:11.575536 26310 ts_tablet_manager.cc:1398] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240502 01:54:11.578055 26308 tablet_bootstrap.cc:492] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:11.578313 26308 ts_tablet_manager.cc:1367] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.060s user 0.000s sys 0.029s
I20240502 01:54:11.579115 22010 tablet_copy_service.cc:140] P 06543b05a16f42099bc86db2a0284412: Received BeginTabletCopySession request for tablet 0f8200b044ef44e1a0451baf2b1485d2 from peer d7449bb1df214bf4a5f93e4631312f08 ({username='slave'} at 127.14.212.199:39887)
I20240502 01:54:11.579360 22010 tablet_copy_service.cc:161] P 06543b05a16f42099bc86db2a0284412: Beginning new tablet copy session on tablet 0f8200b044ef44e1a0451baf2b1485d2 from peer d7449bb1df214bf4a5f93e4631312f08 at {username='slave'} at 127.14.212.199:39887: session id = d7449bb1df214bf4a5f93e4631312f08-0f8200b044ef44e1a0451baf2b1485d2
I20240502 01:54:11.579368 26308 raft_consensus.cc:348] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:11.579818 26308 raft_consensus.cc:374] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.579916 26308 raft_consensus.cc:729] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.580713 26308 consensus_queue.cc:260] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:11.581398 26308 ts_tablet_manager.cc:1398] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:11.583515 22010 tablet_copy_source_session.cc:205] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:11.565685 26303 ts_tablet_manager.cc:1398] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 01:54:11.586310 26303 tablet_bootstrap.cc:492] T 8b377c7402194b42a8eabc747f668fb2 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:11.588861 26303 tablet_bootstrap.cc:654] T 8b377c7402194b42a8eabc747f668fb2 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.592854 26301 tablet_bootstrap.cc:492] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:11.593225 26301 ts_tablet_manager.cc:1367] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.152s user 0.016s sys 0.043s
I20240502 01:54:11.594535 26301 raft_consensus.cc:348] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.594960 26301 raft_consensus.cc:374] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.595199 26301 raft_consensus.cc:729] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.595620 26301 consensus_queue.cc:260] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.597115 26305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8200b044ef44e1a0451baf2b1485d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.597906 26301 ts_tablet_manager.cc:1398] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:54:11.602537 26301 tablet_bootstrap.cc:492] T feffa925599b4d6aafa48e09e4f6467a P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:11.605893 26302 tablet_bootstrap.cc:492] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:11.606125 26302 ts_tablet_manager.cc:1367] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.021s
I20240502 01:54:11.607273 26301 tablet_bootstrap.cc:654] T feffa925599b4d6aafa48e09e4f6467a P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.607178 26302 raft_consensus.cc:348] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.607501 26302 raft_consensus.cc:374] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.607659 26302 raft_consensus.cc:729] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.607991 26302 consensus_queue.cc:260] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.610415 26302 ts_tablet_manager.cc:1398] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:54:11.624779 26303 tablet_bootstrap.cc:492] T 8b377c7402194b42a8eabc747f668fb2 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:11.624958 26305 tablet_copy_client.cc:783] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:11.625136 26303 ts_tablet_manager.cc:1367] T 8b377c7402194b42a8eabc747f668fb2 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.017s
I20240502 01:54:11.625288 26305 tablet_copy_client.cc:647] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:11.626147 26301 tablet_bootstrap.cc:492] T feffa925599b4d6aafa48e09e4f6467a P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:11.626456 26301 ts_tablet_manager.cc:1367] T feffa925599b4d6aafa48e09e4f6467a P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.006s
I20240502 01:54:11.626451 26303 raft_consensus.cc:348] T 8b377c7402194b42a8eabc747f668fb2 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:11.626847 26303 raft_consensus.cc:374] T 8b377c7402194b42a8eabc747f668fb2 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.627002 26303 raft_consensus.cc:729] T 8b377c7402194b42a8eabc747f668fb2 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.627645 26301 raft_consensus.cc:348] T feffa925599b4d6aafa48e09e4f6467a P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.627530 26303 consensus_queue.cc:260] T 8b377c7402194b42a8eabc747f668fb2 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:11.627974 26301 raft_consensus.cc:374] T feffa925599b4d6aafa48e09e4f6467a P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.628080 26301 raft_consensus.cc:729] T feffa925599b4d6aafa48e09e4f6467a P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.628332 26301 consensus_queue.cc:260] T feffa925599b4d6aafa48e09e4f6467a P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.628742 26303 ts_tablet_manager.cc:1398] T 8b377c7402194b42a8eabc747f668fb2 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 01:54:11.629154 26303 tablet_bootstrap.cc:492] T feffa925599b4d6aafa48e09e4f6467a P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:11.630016 26301 ts_tablet_manager.cc:1398] T feffa925599b4d6aafa48e09e4f6467a P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:11.630342 26301 tablet_bootstrap.cc:492] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:11.632043 26301 tablet_bootstrap.cc:654] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.631884 26305 tablet_copy_client.cc:523] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:11.636963 26303 tablet_bootstrap.cc:654] T feffa925599b4d6aafa48e09e4f6467a P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.639124 26305 tablet_bootstrap.cc:492] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:11.653249 24601 raft_consensus.cc:1203] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:11.654274 26303 tablet_bootstrap.cc:492] T feffa925599b4d6aafa48e09e4f6467a P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:11.654498 26303 ts_tablet_manager.cc:1367] T feffa925599b4d6aafa48e09e4f6467a P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.000s
I20240502 01:54:11.655480 26303 raft_consensus.cc:348] T feffa925599b4d6aafa48e09e4f6467a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.656443 26301 tablet_bootstrap.cc:492] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:11.656662 26301 ts_tablet_manager.cc:1367] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.005s
I20240502 01:54:11.657383 26303 raft_consensus.cc:374] T feffa925599b4d6aafa48e09e4f6467a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.657531 26303 raft_consensus.cc:729] T feffa925599b4d6aafa48e09e4f6467a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.657850 26303 consensus_queue.cc:260] T feffa925599b4d6aafa48e09e4f6467a P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.658828 26303 ts_tablet_manager.cc:1398] T feffa925599b4d6aafa48e09e4f6467a P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:54:11.660305 26301 raft_consensus.cc:348] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:11.660547 26301 raft_consensus.cc:374] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.660673 26301 raft_consensus.cc:729] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.660987 26301 consensus_queue.cc:260] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:11.661660 26301 ts_tablet_manager.cc:1398] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:54:11.662961 26301 tablet_bootstrap.cc:492] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:11.664669 26301 tablet_bootstrap.cc:654] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.677438 26301 tablet_bootstrap.cc:492] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:11.677700 26301 ts_tablet_manager.cc:1367] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20240502 01:54:11.678514 26301 raft_consensus.cc:348] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:11.678812 26301 raft_consensus.cc:374] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.678931 26301 raft_consensus.cc:729] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.679211 26301 consensus_queue.cc:260] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:11.679769 26301 ts_tablet_manager.cc:1398] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:11.698329 26041 raft_consensus.cc:482] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:11.698575 26041 raft_consensus.cc:504] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:11.699290 26041 leader_election.cc:288] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:11.700364 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b377c7402194b42a8eabc747f668fb2" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:11.700600 24868 raft_consensus.cc:2454] T 8b377c7402194b42a8eabc747f668fb2 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:11.701100 24669 leader_election.cc:302] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:11.701128 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b377c7402194b42a8eabc747f668fb2" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:11.701364 24601 raft_consensus.cc:2454] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:11.701465 26041 raft_consensus.cc:2781] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:11.701546 26041 raft_consensus.cc:482] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:11.701666 26041 raft_consensus.cc:3037] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.703564 26305 tablet_bootstrap.cc:492] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:11.703601 26041 raft_consensus.cc:504] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:11.704257 26041 leader_election.cc:288] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:11.704408 26305 tablet_bootstrap.cc:492] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap complete.
I20240502 01:54:11.704728 26305 ts_tablet_manager.cc:1367] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.066s user 0.046s sys 0.004s
I20240502 01:54:11.705802 26305 raft_consensus.cc:348] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.706185 26305 raft_consensus.cc:729] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.706703 26305 consensus_queue.cc:260] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.708202 26305 ts_tablet_manager.cc:1398] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:11.706280 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b377c7402194b42a8eabc747f668fb2" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:11.708599 24868 raft_consensus.cc:3037] T 8b377c7402194b42a8eabc747f668fb2 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.709328 22010 tablet_copy_service.cc:342] P 06543b05a16f42099bc86db2a0284412: Request end of tablet copy session d7449bb1df214bf4a5f93e4631312f08-0f8200b044ef44e1a0451baf2b1485d2 received from {username='slave'} at 127.14.212.199:39887
I20240502 01:54:11.709463 22010 tablet_copy_service.cc:433] P 06543b05a16f42099bc86db2a0284412: ending tablet copy session d7449bb1df214bf4a5f93e4631312f08-0f8200b044ef44e1a0451baf2b1485d2 on tablet 0f8200b044ef44e1a0451baf2b1485d2 with peer d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:11.711447 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b377c7402194b42a8eabc747f668fb2" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:11.711691 24601 raft_consensus.cc:3037] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.711928 24868 raft_consensus.cc:2454] T 8b377c7402194b42a8eabc747f668fb2 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:11.712455 24669 leader_election.cc:302] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:11.713191 26041 raft_consensus.cc:2781] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:11.714260 26041 raft_consensus.cc:686] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:11.714625 26041 consensus_queue.cc:237] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:11.715458 24601 raft_consensus.cc:2454] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:11.719110 21714 catalog_manager.cc:5484] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:11.756959 26354 ts_tablet_manager.cc:902] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f: Initiating tablet copy from peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:11.777563 26354 tablet_copy_client.cc:308] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.195:38879
I20240502 01:54:11.795579 22140 tablet_copy_service.cc:140] P c64840657cdf4b93980bd34120c94d9a: Received BeginTabletCopySession request for tablet 9bceab60366b4ade98f86072d391e7f4 from peer 0c64d40b5d7644bd8b29feaa80eab72f ({username='slave'} at 127.14.212.197:58377)
I20240502 01:54:11.795823 22140 tablet_copy_service.cc:161] P c64840657cdf4b93980bd34120c94d9a: Beginning new tablet copy session on tablet 9bceab60366b4ade98f86072d391e7f4 from peer 0c64d40b5d7644bd8b29feaa80eab72f at {username='slave'} at 127.14.212.197:58377: session id = 0c64d40b5d7644bd8b29feaa80eab72f-9bceab60366b4ade98f86072d391e7f4
I20240502 01:54:11.799049 22140 tablet_copy_source_session.cc:205] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:11.806582 26354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bceab60366b4ade98f86072d391e7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.817405 26354 tablet_copy_client.cc:783] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:11.817809 26354 tablet_copy_client.cc:647] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:11.828482 26354 tablet_copy_client.cc:523] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:11.856715 26354 tablet_bootstrap.cc:492] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:11.907487 26258 raft_consensus.cc:482] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:11.907711 26258 raft_consensus.cc:504] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:11.908455 26258 leader_election.cc:288] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:11.910938 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149912f54ab74daf97652148e4ede2be" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:11.911226 24601 raft_consensus.cc:2454] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:54:11.914733 26313 raft_consensus.cc:482] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:11.914909 26313 raft_consensus.cc:504] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.915661 26313 leader_election.cc:288] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:11.915948 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149912f54ab74daf97652148e4ede2be" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:11.916191 22249 raft_consensus.cc:2454] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:54:11.916270 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feffa925599b4d6aafa48e09e4f6467a" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:11.916514 24868 raft_consensus.cc:2454] T feffa925599b4d6aafa48e09e4f6467a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:11.917220 22054 leader_election.cc:302] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:54:11.919539 24934 leader_election.cc:302] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:11.920320 25400 raft_consensus.cc:2781] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:11.920429 25400 raft_consensus.cc:482] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:11.920529 25400 raft_consensus.cc:3037] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.930262 25400 raft_consensus.cc:504] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:11.931106 24868 raft_consensus.cc:1203] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:11.932832 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feffa925599b4d6aafa48e09e4f6467a" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:11.933145 24601 raft_consensus.cc:2454] T feffa925599b4d6aafa48e09e4f6467a P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:11.935364 26313 raft_consensus.cc:2781] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:11.935513 26313 raft_consensus.cc:482] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:11.935623 26313 raft_consensus.cc:3037] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.938535 26313 raft_consensus.cc:504] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.939312 26313 leader_election.cc:288] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:11.940062 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feffa925599b4d6aafa48e09e4f6467a" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:11.940269 24868 raft_consensus.cc:3037] T feffa925599b4d6aafa48e09e4f6467a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.940335 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feffa925599b4d6aafa48e09e4f6467a" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:11.940606 24601 raft_consensus.cc:3037] T feffa925599b4d6aafa48e09e4f6467a P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.942793 24868 raft_consensus.cc:2454] T feffa925599b4d6aafa48e09e4f6467a P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:11.946496 24601 raft_consensus.cc:2454] T feffa925599b4d6aafa48e09e4f6467a P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:11.947284 26354 tablet_bootstrap.cc:492] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:11.947798 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149912f54ab74daf97652148e4ede2be" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:11.947952 25400 leader_election.cc:288] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:11.948009 24601 raft_consensus.cc:3037] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.948025 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149912f54ab74daf97652148e4ede2be" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:11.943329 24934 leader_election.cc:302] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:11.948375 22249 raft_consensus.cc:3037] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.948844 26313 raft_consensus.cc:2781] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:11.949414 26313 raft_consensus.cc:686] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:54:11.949801 26313 consensus_queue.cc:237] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.950660 24601 raft_consensus.cc:2454] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:54:11.951244 22249 raft_consensus.cc:2454] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:54:11.951251 22054 leader_election.cc:302] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:54:11.951700 25400 raft_consensus.cc:2781] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:11.951953 25400 raft_consensus.cc:686] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:54:11.947806 26354 tablet_bootstrap.cc:492] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap complete.
I20240502 01:54:11.952620 26354 ts_tablet_manager.cc:1367] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.096s user 0.058s sys 0.026s
I20240502 01:54:11.952461 25400 consensus_queue.cc:237] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:11.953701 26354 raft_consensus.cc:348] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:11.953972 26354 raft_consensus.cc:729] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.954206 26354 consensus_queue.cc:260] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:11.954840 26354 ts_tablet_manager.cc:1398] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:54:11.955860 22140 tablet_copy_service.cc:342] P c64840657cdf4b93980bd34120c94d9a: Request end of tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-9bceab60366b4ade98f86072d391e7f4 received from {username='slave'} at 127.14.212.197:58377
I20240502 01:54:11.955986 22140 tablet_copy_service.cc:433] P c64840657cdf4b93980bd34120c94d9a: ending tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-9bceab60366b4ade98f86072d391e7f4 on tablet 9bceab60366b4ade98f86072d391e7f4 with peer 0c64d40b5d7644bd8b29feaa80eab72f
I20240502 01:54:11.961154 21712 catalog_manager.cc:5484] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:11.963219 26215 raft_consensus.cc:482] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:11.963389 26215 raft_consensus.cc:504] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.964116 26215 leader_election.cc:288] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:11.965350 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a16aff54e6f4975a97776c214890f57" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:11.965623 21857 raft_consensus.cc:2454] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:11.966404 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a16aff54e6f4975a97776c214890f57" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:11.966502 24801 leader_election.cc:302] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:11.966678 24601 raft_consensus.cc:2454] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:11.966641 21714 catalog_manager.cc:5484] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:11.967445 26215 raft_consensus.cc:2781] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:11.967566 26215 raft_consensus.cc:482] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:11.967659 26215 raft_consensus.cc:3037] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.970369 26215 raft_consensus.cc:504] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.970968 26215 leader_election.cc:288] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:11.971496 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a16aff54e6f4975a97776c214890f57" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:11.971737 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a16aff54e6f4975a97776c214890f57" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:11.971837 24601 raft_consensus.cc:3037] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.971999 21857 raft_consensus.cc:3037] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.975082 21857 raft_consensus.cc:2454] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:11.975479 24601 raft_consensus.cc:2454] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:11.976751 24801 leader_election.cc:302] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:11.977159 26215 raft_consensus.cc:2781] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:11.977676 26215 raft_consensus.cc:686] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:11.978013 26215 consensus_queue.cc:237] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:11.984635 21714 catalog_manager.cc:5484] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:12.036164 21967 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:12.055406 24583 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:54:12.075029 24979 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:12.077772 24714 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:12.087136 22231 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:12.089679 21836 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:12.093905 21714 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:12.106592 22098 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:12.113632 26041 consensus_queue.cc:1035] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:12.117872 24603 raft_consensus.cc:1261] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:12.118628 26258 consensus_queue.cc:1035] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:12.119364 22250 raft_consensus.cc:1261] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:12.124913 26258 consensus_queue.cc:1035] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:12.178670 21714 catalog_manager.cc:5576] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093) does not have the latest schema for tablet 149912f54ab74daf97652148e4ede2be (table rebalancer_dynamic_table [id=ca8fb519fdb343f09b50f29586ecfb30]). Expected version 1 got 0
I20240502 01:54:12.195437 24848 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:12.210613 24602 raft_consensus.cc:1203] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:12.214342 26393 mvcc.cc:205] Tried to move back new op lower bound from 7023062434261450752 to 7023062433601929216. Current Snapshot: MvccSnapshot[applied={T|T < 7023062434261450752}]
I20240502 01:54:12.214680 26401 tablet.cc:1722] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:12.216212 26401 tablet.cc:1620] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a: MemRowSet was empty: no flush needed.
I20240502 01:54:12.219249 21713 catalog_manager.cc:5576] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677) does not have the latest schema for tablet 149912f54ab74daf97652148e4ede2be (table rebalancer_dynamic_table [id=ca8fb519fdb343f09b50f29586ecfb30]). Expected version 1 got 0
I20240502 01:54:12.220461 26159 consensus_queue.cc:1035] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20240502 01:54:12.231941 21713 catalog_manager.cc:5576] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093) does not have the latest schema for tablet 149912f54ab74daf97652148e4ede2be (table rebalancer_dynamic_table [id=ca8fb519fdb343f09b50f29586ecfb30]). Expected version 1 got 0
I20240502 01:54:12.233481 26402 tablet.cc:1722] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:12.237557 26403 tablet.cc:1722] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:12.238128 26402 tablet.cc:1620] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff: MemRowSet was empty: no flush needed.
I20240502 01:54:12.239972 21714 catalog_manager.cc:5576] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677) does not have the latest schema for tablet 149912f54ab74daf97652148e4ede2be (table rebalancer_dynamic_table [id=ca8fb519fdb343f09b50f29586ecfb30]). Expected version 1 got 0
I20240502 01:54:12.253103 21713 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=ca8fb519fdb343f09b50f29586ecfb30] alter complete (version 1)
I20240502 01:54:12.254701 26403 tablet.cc:1620] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:12.327692 26313 consensus_queue.cc:1035] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:12.360908 21713 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:12.362358 26187 consensus_queue.cc:1035] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:12.375052 26401 tablet.cc:1722] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:12.375617 26401 tablet.cc:1620] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a: MemRowSet was empty: no flush needed.
I20240502 01:54:12.375830 26403 tablet.cc:1722] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:12.376312 26403 tablet.cc:1620] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:12.381026 26402 tablet.cc:1722] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:12.381565 26402 tablet.cc:1620] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff: MemRowSet was empty: no flush needed.
I20240502 01:54:12.387434 21713 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=ca8fb519fdb343f09b50f29586ecfb30] alter complete (version 2)
I20240502 01:54:12.440577 21988 consensus_queue.cc:237] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:12.442709 21857 raft_consensus.cc:1261] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:12.442728 24867 raft_consensus.cc:1261] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:12.443316 26175 consensus_queue.cc:1035] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:12.443626 26182 consensus_queue.cc:1035] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:12.446875 26175 raft_consensus.cc:2932] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:12.448052 24867 raft_consensus.cc:2932] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:12.449824 21857 raft_consensus.cc:2932] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:12.454216 21714 catalog_manager.cc:5484] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:12.459798 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f8200b044ef44e1a0451baf2b1485d2 with delete_type TABLET_DATA_TOMBSTONED (TS c2a28baa7ee2469294d94b7fe5455cff not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:12.473146 26430 tablet_replica.cc:330] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:12.473376 26430 raft_consensus.cc:2229] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:12.473722 26430 raft_consensus.cc:2258] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:12.474576 26430 ts_tablet_manager.cc:1875] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:12.481006 26430 ts_tablet_manager.cc:1888] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:54:12.481227 26430 log.cc:1196] T 0f8200b044ef44e1a0451baf2b1485d2 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/0f8200b044ef44e1a0451baf2b1485d2
I20240502 01:54:12.482064 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 0f8200b044ef44e1a0451baf2b1485d2 (table rebalance_test_table_17 [id=7a17c6db79004eb696e7dd281fdc4f3b]) successfully deleted
I20240502 01:54:12.491801 21712 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:12.500460 26401 tablet.cc:1722] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:12.501858 26402 tablet.cc:1722] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:12.502620 26215 consensus_queue.cc:1035] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:12.508496 26403 tablet.cc:1722] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:12.516340 21712 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=ca8fb519fdb343f09b50f29586ecfb30] alter complete (version 3)
I20240502 01:54:12.518170 26322 consensus_queue.cc:1035] T 5a16aff54e6f4975a97776c214890f57 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:12.540355 21713 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:12.552733 24979 tablet_service.cc:1467] Processing CreateTablet for tablet 6dad8caf4f9744eea7be75d0a4890f65 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=869549e86fb24c718467c2625a66e8c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.552747 24714 tablet_service.cc:1467] Processing CreateTablet for tablet 6dad8caf4f9744eea7be75d0a4890f65 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=869549e86fb24c718467c2625a66e8c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.552942 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 6dad8caf4f9744eea7be75d0a4890f65 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=869549e86fb24c718467c2625a66e8c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.553303 24714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dad8caf4f9744eea7be75d0a4890f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.553323 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dad8caf4f9744eea7be75d0a4890f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.553372 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dad8caf4f9744eea7be75d0a4890f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.555291 24582 tablet_service.cc:1467] Processing CreateTablet for tablet 752d84f8d70e4bed91d0ea547923f22f (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=869549e86fb24c718467c2625a66e8c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.555749 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752d84f8d70e4bed91d0ea547923f22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.556484 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4e454146f24104adf99041a0e5a3ad (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=869549e86fb24c718467c2625a66e8c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.556895 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e4e454146f24104adf99041a0e5a3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.557127 24581 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4e454146f24104adf99041a0e5a3ad (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=869549e86fb24c718467c2625a66e8c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.557518 24581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e4e454146f24104adf99041a0e5a3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.562192 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 752d84f8d70e4bed91d0ea547923f22f (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=869549e86fb24c718467c2625a66e8c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.562280 24713 tablet_service.cc:1467] Processing CreateTablet for tablet 752d84f8d70e4bed91d0ea547923f22f (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=869549e86fb24c718467c2625a66e8c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.562698 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752d84f8d70e4bed91d0ea547923f22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.562696 24713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752d84f8d70e4bed91d0ea547923f22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.565389 24981 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4e454146f24104adf99041a0e5a3ad (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=869549e86fb24c718467c2625a66e8c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.565868 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e4e454146f24104adf99041a0e5a3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.583909 26449 tablet_bootstrap.cc:492] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:12.584524 26451 tablet_bootstrap.cc:492] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:12.589505 26449 tablet_bootstrap.cc:654] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.589563 26451 tablet_bootstrap.cc:654] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.592134 26450 tablet_bootstrap.cc:492] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:12.594688 26450 tablet_bootstrap.cc:654] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.597047 26448 tablet_bootstrap.cc:492] T 6e4e454146f24104adf99041a0e5a3ad P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:12.599013 26449 tablet_bootstrap.cc:492] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:12.599258 26449 ts_tablet_manager.cc:1367] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20240502 01:54:12.599392 26448 tablet_bootstrap.cc:654] T 6e4e454146f24104adf99041a0e5a3ad P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.600394 26449 raft_consensus.cc:348] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:12.601050 26449 raft_consensus.cc:374] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.601192 26449 raft_consensus.cc:729] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.601449 26450 tablet_bootstrap.cc:492] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:12.601687 26450 ts_tablet_manager.cc:1367] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.001s
I20240502 01:54:12.601559 26449 consensus_queue.cc:260] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:12.602281 26449 ts_tablet_manager.cc:1398] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:12.602523 26450 raft_consensus.cc:348] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:12.602790 26450 raft_consensus.cc:374] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.602946 26450 raft_consensus.cc:729] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.603246 26450 consensus_queue.cc:260] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:12.603368 26449 tablet_bootstrap.cc:492] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:12.605577 26451 tablet_bootstrap.cc:492] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:12.605836 26451 ts_tablet_manager.cc:1367] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.000s
I20240502 01:54:12.606426 26450 ts_tablet_manager.cc:1398] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:54:12.606803 26450 tablet_bootstrap.cc:492] T 6e4e454146f24104adf99041a0e5a3ad P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:12.607930 26449 tablet_bootstrap.cc:654] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.608661 26450 tablet_bootstrap.cc:654] T 6e4e454146f24104adf99041a0e5a3ad P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.608716 26197 raft_consensus.cc:482] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:12.613783 26451 raft_consensus.cc:348] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:12.614084 26451 raft_consensus.cc:374] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.614181 26451 raft_consensus.cc:729] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.614459 26451 consensus_queue.cc:260] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:12.614831 26197 raft_consensus.cc:504] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:12.615886 26197 leader_election.cc:288] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:12.616835 26448 tablet_bootstrap.cc:492] T 6e4e454146f24104adf99041a0e5a3ad P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:12.616961 26451 ts_tablet_manager.cc:1398] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:54:12.617049 26448 ts_tablet_manager.cc:1367] T 6e4e454146f24104adf99041a0e5a3ad P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.005s
I20240502 01:54:12.617255 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4e454146f24104adf99041a0e5a3ad" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:12.617956 26448 raft_consensus.cc:348] T 6e4e454146f24104adf99041a0e5a3ad P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:12.618202 26448 raft_consensus.cc:374] T 6e4e454146f24104adf99041a0e5a3ad P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.618307 26448 raft_consensus.cc:729] T 6e4e454146f24104adf99041a0e5a3ad P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
W20240502 01:54:12.618466 24539 leader_election.cc:341] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:12.618566 26448 consensus_queue.cc:260] T 6e4e454146f24104adf99041a0e5a3ad P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:12.619303 26448 ts_tablet_manager.cc:1398] T 6e4e454146f24104adf99041a0e5a3ad P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:12.619602 26448 tablet_bootstrap.cc:492] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:12.620976 26448 tablet_bootstrap.cc:654] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.621325 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4e454146f24104adf99041a0e5a3ad" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:12.625126 24735 raft_consensus.cc:2454] T 6e4e454146f24104adf99041a0e5a3ad P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:12.625769 24537 leader_election.cc:302] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters: 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:12.627151 26448 tablet_bootstrap.cc:492] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:12.627750 26448 ts_tablet_manager.cc:1367] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 01:54:12.627763 21713 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:12.627976 21713 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:12.628835 26448 raft_consensus.cc:348] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:12.629081 26448 raft_consensus.cc:374] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.629159 26448 raft_consensus.cc:729] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.629416 26448 consensus_queue.cc:260] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:12.630074 26448 ts_tablet_manager.cc:1398] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:12.630369 26448 tablet_bootstrap.cc:492] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:12.632514 26448 tablet_bootstrap.cc:654] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.634950 26449 tablet_bootstrap.cc:492] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:12.635156 26449 ts_tablet_manager.cc:1367] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.003s
I20240502 01:54:12.635713 21713 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 149912f54ab74daf97652148e4ede2be
I20240502 01:54:12.636000 26449 raft_consensus.cc:348] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:12.636260 26449 raft_consensus.cc:374] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.636343 26449 raft_consensus.cc:729] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.636615 26449 consensus_queue.cc:260] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:12.636862 24582 tablet_service.cc:1514] Processing DeleteTablet for tablet 149912f54ab74daf97652148e4ede2be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:12 UTC) from {username='slave'} at 127.0.0.1:53832
I20240502 01:54:12.637341 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet 149912f54ab74daf97652148e4ede2be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:12 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:12.637596 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 149912f54ab74daf97652148e4ede2be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:12 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:12.637709 26430 tablet_replica.cc:330] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:12.637998 26430 raft_consensus.cc:2229] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:12.638314 26430 raft_consensus.cc:2258] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:12.639990 26430 ts_tablet_manager.cc:1875] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:12.640535 26463 tablet_replica.cc:330] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:54:12.640895 26463 raft_consensus.cc:2229] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:12.641194 26463 raft_consensus.cc:2258] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:12.642336 26463 ts_tablet_manager.cc:1875] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:12.649241 26197 raft_consensus.cc:2781] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:12.649456 26197 raft_consensus.cc:482] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:12.649580 26197 raft_consensus.cc:3037] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.650998 26463 ts_tablet_manager.cc:1888] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:12.651202 26463 log.cc:1196] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/149912f54ab74daf97652148e4ede2be
I20240502 01:54:12.651865 26463 ts_tablet_manager.cc:1909] T 149912f54ab74daf97652148e4ede2be P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting consensus metadata
I20240502 01:54:12.653367 26464 tablet_replica.cc:330] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:12.653788 26464 raft_consensus.cc:2229] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:12.654657 26041 raft_consensus.cc:482] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:12.654919 26041 raft_consensus.cc:504] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:12.654343 26464 raft_consensus.cc:2258] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:12.655918 26041 leader_election.cc:288] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:12.656916 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dad8caf4f9744eea7be75d0a4890f65" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:12.657174 26464 ts_tablet_manager.cc:1875] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:12.657271 24601 raft_consensus.cc:2454] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:12.658445 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dad8caf4f9744eea7be75d0a4890f65" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:12.658823 25001 raft_consensus.cc:2454] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:12.659674 24671 leader_election.cc:302] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:12.662747 26449 ts_tablet_manager.cc:1398] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.026s user 0.001s sys 0.012s
I20240502 01:54:12.663148 26449 tablet_bootstrap.cc:492] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:12.663872 26041 raft_consensus.cc:2781] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:12.664011 26041 raft_consensus.cc:482] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:12.664135 26041 raft_consensus.cc:3037] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.665990 26449 tablet_bootstrap.cc:654] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.670522 26464 ts_tablet_manager.cc:1888] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:12.670965 26464 log.cc:1196] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/149912f54ab74daf97652148e4ede2be
I20240502 01:54:12.671494 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet 149912f54ab74daf97652148e4ede2be (table rebalancer_dynamic_table_new_name [id=ca8fb519fdb343f09b50f29586ecfb30]) successfully deleted
I20240502 01:54:12.671590 26464 ts_tablet_manager.cc:1909] T 149912f54ab74daf97652148e4ede2be P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:12.672621 26449 tablet_bootstrap.cc:492] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:12.673499 26448 tablet_bootstrap.cc:492] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:12.673730 26448 ts_tablet_manager.cc:1367] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.006s
I20240502 01:54:12.673785 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 149912f54ab74daf97652148e4ede2be (table rebalancer_dynamic_table_new_name [id=ca8fb519fdb343f09b50f29586ecfb30]) successfully deleted
I20240502 01:54:12.675652 26448 raft_consensus.cc:348] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:12.675976 26448 raft_consensus.cc:374] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.676142 26448 raft_consensus.cc:729] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.676501 26448 consensus_queue.cc:260] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:12.677340 26448 ts_tablet_manager.cc:1398] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:12.677882 26449 ts_tablet_manager.cc:1367] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.002s
I20240502 01:54:12.678686 26449 raft_consensus.cc:348] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:12.678871 26449 raft_consensus.cc:374] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.678968 26449 raft_consensus.cc:729] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.679028 26041 raft_consensus.cc:504] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:12.679350 26449 consensus_queue.cc:260] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:12.681897 26449 ts_tablet_manager.cc:1398] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:54:12.691987 26197 raft_consensus.cc:504] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:12.693809 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4e454146f24104adf99041a0e5a3ad" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:12.694150 24735 raft_consensus.cc:3037] T 6e4e454146f24104adf99041a0e5a3ad P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.701097 24735 raft_consensus.cc:2454] T 6e4e454146f24104adf99041a0e5a3ad P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:12.701853 24537 leader_election.cc:302] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:54:12.702220 26462 raft_consensus.cc:2781] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:12.702752 26462 raft_consensus.cc:686] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:54:12.703123 26462 consensus_queue.cc:237] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:12.705919 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4e454146f24104adf99041a0e5a3ad" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
W20240502 01:54:12.706919 24539 leader_election.cc:341] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:12.707078 21713 catalog_manager.cc:5484] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:12.707753 26197 leader_election.cc:288] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:12.709812 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dad8caf4f9744eea7be75d0a4890f65" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:12.710150 25001 raft_consensus.cc:3037] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.711638 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dad8caf4f9744eea7be75d0a4890f65" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:12.711930 24601 raft_consensus.cc:3037] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.716127 24601 raft_consensus.cc:2454] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:12.716774 25001 raft_consensus.cc:2454] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:12.716774 24669 leader_election.cc:302] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:54:12.721212 26159 raft_consensus.cc:2781] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:12.721493 26041 leader_election.cc:288] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:12.725234 26430 ts_tablet_manager.cc:1888] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:12.725512 26430 log.cc:1196] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/149912f54ab74daf97652148e4ede2be
I20240502 01:54:12.726171 26430 ts_tablet_manager.cc:1909] T 149912f54ab74daf97652148e4ede2be P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:12.727815 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 149912f54ab74daf97652148e4ede2be (table rebalancer_dynamic_table_new_name [id=ca8fb519fdb343f09b50f29586ecfb30]) successfully deleted
I20240502 01:54:12.728921 26450 tablet_bootstrap.cc:492] T 6e4e454146f24104adf99041a0e5a3ad P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:12.729133 26450 ts_tablet_manager.cc:1367] T 6e4e454146f24104adf99041a0e5a3ad P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.122s user 0.051s sys 0.052s
I20240502 01:54:12.729913 26450 raft_consensus.cc:348] T 6e4e454146f24104adf99041a0e5a3ad P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:12.730077 26159 raft_consensus.cc:686] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:12.730115 26450 raft_consensus.cc:374] T 6e4e454146f24104adf99041a0e5a3ad P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.730187 26450 raft_consensus.cc:729] T 6e4e454146f24104adf99041a0e5a3ad P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.730427 26450 consensus_queue.cc:260] T 6e4e454146f24104adf99041a0e5a3ad P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:12.730455 26159 consensus_queue.cc:237] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:12.730924 26450 ts_tablet_manager.cc:1398] T 6e4e454146f24104adf99041a0e5a3ad P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:12.734964 21714 catalog_manager.cc:5484] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:12.763470 21715 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:12.776645 24979 tablet_service.cc:1467] Processing CreateTablet for tablet b3f8e9fb62f44e4d93a5183f5c137ef0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6ea7ec30ec06464fa77ecc96efb15aaf]), partition=
I20240502 01:54:12.777107 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3f8e9fb62f44e4d93a5183f5c137ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.777714 24582 tablet_service.cc:1467] Processing CreateTablet for tablet b3f8e9fb62f44e4d93a5183f5c137ef0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6ea7ec30ec06464fa77ecc96efb15aaf]), partition=
I20240502 01:54:12.777752 24715 tablet_service.cc:1467] Processing CreateTablet for tablet b3f8e9fb62f44e4d93a5183f5c137ef0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6ea7ec30ec06464fa77ecc96efb15aaf]), partition=
I20240502 01:54:12.778128 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3f8e9fb62f44e4d93a5183f5c137ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.778138 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3f8e9fb62f44e4d93a5183f5c137ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.783074 26449 tablet_bootstrap.cc:492] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:12.785586 26449 tablet_bootstrap.cc:654] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.786592 26450 tablet_bootstrap.cc:492] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:12.788168 26450 tablet_bootstrap.cc:654] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.788360 26449 tablet_bootstrap.cc:492] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:12.788535 26449 ts_tablet_manager.cc:1367] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:54:12.789381 26449 raft_consensus.cc:348] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:12.789655 26449 raft_consensus.cc:374] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.789768 26449 raft_consensus.cc:729] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.790051 26449 consensus_queue.cc:260] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:12.790661 26450 tablet_bootstrap.cc:492] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:12.790840 26450 ts_tablet_manager.cc:1367] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 01:54:12.791599 26450 raft_consensus.cc:348] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:12.791826 26450 raft_consensus.cc:374] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.791932 26450 raft_consensus.cc:729] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.792169 26450 consensus_queue.cc:260] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:12.792781 26450 ts_tablet_manager.cc:1398] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:12.793260 26449 ts_tablet_manager.cc:1398] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:54:12.802913 26448 tablet_bootstrap.cc:492] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:12.804153 26159 raft_consensus.cc:482] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:12.804358 26159 raft_consensus.cc:504] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:12.805100 26159 leader_election.cc:288] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:12.805456 26448 tablet_bootstrap.cc:654] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.806840 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752d84f8d70e4bed91d0ea547923f22f" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:12.807163 22250 raft_consensus.cc:2454] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:12.807641 24670 leader_election.cc:302] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:12.807688 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752d84f8d70e4bed91d0ea547923f22f" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:12.807907 26159 raft_consensus.cc:2781] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:12.807997 26159 raft_consensus.cc:482] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:12.808110 26159 raft_consensus.cc:3037] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.808063 24601 raft_consensus.cc:2454] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:12.811117 26159 raft_consensus.cc:504] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:12.812069 26159 leader_election.cc:288] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:12.812536 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752d84f8d70e4bed91d0ea547923f22f" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:12.812788 24601 raft_consensus.cc:3037] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.815140 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752d84f8d70e4bed91d0ea547923f22f" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:12.815380 22250 raft_consensus.cc:3037] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.819825 22250 raft_consensus.cc:2454] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:12.821274 26408 raft_consensus.cc:482] T 6e4e454146f24104adf99041a0e5a3ad P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:12.821496 24601 raft_consensus.cc:2454] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:12.821437 26408 raft_consensus.cc:504] T 6e4e454146f24104adf99041a0e5a3ad P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:12.822145 26408 leader_election.cc:288] T 6e4e454146f24104adf99041a0e5a3ad P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:12.822468 24670 leader_election.cc:302] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:12.823303 26159 raft_consensus.cc:2781] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:12.823482 26159 raft_consensus.cc:686] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:12.823745 26159 consensus_queue.cc:237] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:12.827832 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4e454146f24104adf99041a0e5a3ad" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:12.827916 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4e454146f24104adf99041a0e5a3ad" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:12.828181 24735 raft_consensus.cc:2379] T 6e4e454146f24104adf99041a0e5a3ad P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 41c70f373c9d49a49891b03a12c6cb2b in current term 1: Already voted for candidate 0c64d40b5d7644bd8b29feaa80eab72f in this term.
I20240502 01:54:12.833982 21712 catalog_manager.cc:5484] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:12.835064 24934 leader_election.cc:302] T 6e4e454146f24104adf99041a0e5a3ad P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513
I20240502 01:54:12.836033 26313 raft_consensus.cc:3037] T 6e4e454146f24104adf99041a0e5a3ad P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.839792 26313 raft_consensus.cc:2726] T 6e4e454146f24104adf99041a0e5a3ad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:12.870980 26448 tablet_bootstrap.cc:492] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:12.871280 26448 ts_tablet_manager.cc:1367] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.068s user 0.005s sys 0.005s
I20240502 01:54:12.872177 26448 raft_consensus.cc:348] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:12.873128 26448 raft_consensus.cc:374] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.873314 26448 raft_consensus.cc:729] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.874727 26448 consensus_queue.cc:260] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:12.879307 26448 ts_tablet_manager.cc:1398] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:54:12.903499 26408 raft_consensus.cc:482] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:12.903693 26408 raft_consensus.cc:504] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:12.904405 26408 leader_election.cc:288] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:12.905958 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f8e9fb62f44e4d93a5183f5c137ef0" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:12.906217 24601 raft_consensus.cc:2454] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:12.907219 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f8e9fb62f44e4d93a5183f5c137ef0" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:12.907449 24735 raft_consensus.cc:2454] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:12.908093 24934 leader_election.cc:302] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:12.910356 26408 raft_consensus.cc:2781] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:12.910475 26408 raft_consensus.cc:482] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:12.910606 26408 raft_consensus.cc:3037] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.914180 26408 raft_consensus.cc:504] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:12.915838 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f8e9fb62f44e4d93a5183f5c137ef0" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:12.916110 24601 raft_consensus.cc:3037] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.919168 24601 raft_consensus.cc:2454] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:12.919965 24934 leader_election.cc:302] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:12.920666 26313 raft_consensus.cc:2781] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:12.920781 26313 raft_consensus.cc:686] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:54:12.921209 26313 consensus_queue.cc:237] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:12.933714 26408 leader_election.cc:288] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:12.957899 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f8e9fb62f44e4d93a5183f5c137ef0" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:12.958153 24735 raft_consensus.cc:3037] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.963210 21710 catalog_manager.cc:5484] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:12.971122 24735 raft_consensus.cc:2454] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:12.979104 21967 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:13.058678 24715 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:13.081364 21710 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:13.084036 24582 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:13.101140 26197 consensus_queue.cc:1035] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.102252 24735 raft_consensus.cc:1261] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:13.103257 26408 consensus_queue.cc:1035] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.111961 24601 raft_consensus.cc:1261] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:13.113324 26408 consensus_queue.cc:1035] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.114984 21836 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:13.114909 26159 consensus_queue.cc:1035] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.139995 21710 catalog_manager.cc:5576] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375) does not have the latest schema for tablet b3f8e9fb62f44e4d93a5183f5c137ef0 (table rebalancer_dynamic_table [id=6ea7ec30ec06464fa77ecc96efb15aaf]). Expected version 1 got 0
I20240502 01:54:13.148595 26524 tablet.cc:1722] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:13.149298 26524 tablet.cc:1620] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:13.150877 26525 tablet.cc:1722] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:13.151741 26525 tablet.cc:1620] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:13.158612 21710 catalog_manager.cc:5576] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375) does not have the latest schema for tablet b3f8e9fb62f44e4d93a5183f5c137ef0 (table rebalancer_dynamic_table [id=6ea7ec30ec06464fa77ecc96efb15aaf]). Expected version 1 got 0
I20240502 01:54:13.169891 26462 consensus_queue.cc:1035] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.175908 21710 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=6ea7ec30ec06464fa77ecc96efb15aaf] alter complete (version 1)
I20240502 01:54:13.181622 24979 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:13.183092 26041 consensus_queue.cc:1035] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.186476 21710 catalog_manager.cc:5576] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677) does not have the latest schema for tablet b3f8e9fb62f44e4d93a5183f5c137ef0 (table rebalancer_dynamic_table [id=6ea7ec30ec06464fa77ecc96efb15aaf]). Expected version 1 got 0
I20240502 01:54:13.197422 26531 tablet.cc:1722] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:13.202458 21710 catalog_manager.cc:5576] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677) does not have the latest schema for tablet b3f8e9fb62f44e4d93a5183f5c137ef0 (table rebalancer_dynamic_table [id=6ea7ec30ec06464fa77ecc96efb15aaf]). Expected version 1 got 0
I20240502 01:54:13.202951 26531 tablet.cc:1620] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:13.209081 22231 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:13.235167 22100 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:13.262120 26041 consensus_queue.cc:1035] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.294979 21710 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:13.296296 26041 consensus_queue.cc:1035] T 752d84f8d70e4bed91d0ea547923f22f P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.305651 26524 tablet.cc:1722] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:13.306224 26524 tablet.cc:1620] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:13.308521 26525 tablet.cc:1722] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:13.309083 26525 tablet.cc:1620] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:13.309908 26531 tablet.cc:1722] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:13.310385 26531 tablet.cc:1620] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:13.321076 21710 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=6ea7ec30ec06464fa77ecc96efb15aaf] alter complete (version 2)
I20240502 01:54:13.360582 21709 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:13.372279 24582 tablet_service.cc:1467] Processing CreateTablet for tablet 42957eb0c3864381b67438035ec91c28 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=adc0aa06508a406b91ce8b5a5a44ea29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:13.372829 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42957eb0c3864381b67438035ec91c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.374323 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce01a3c9d9948ba8f153ad150cf3cb0 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=adc0aa06508a406b91ce8b5a5a44ea29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:13.374967 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce01a3c9d9948ba8f153ad150cf3cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.376422 21967 tablet_service.cc:1467] Processing CreateTablet for tablet bddb4a31d81d44afaabc247fd1449767 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=adc0aa06508a406b91ce8b5a5a44ea29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:13.376870 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bddb4a31d81d44afaabc247fd1449767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.376911 24979 tablet_service.cc:1467] Processing CreateTablet for tablet 42957eb0c3864381b67438035ec91c28 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=adc0aa06508a406b91ce8b5a5a44ea29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:13.377336 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42957eb0c3864381b67438035ec91c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.382030 24847 tablet_service.cc:1467] Processing CreateTablet for tablet bddb4a31d81d44afaabc247fd1449767 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=adc0aa06508a406b91ce8b5a5a44ea29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:13.382478 24847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bddb4a31d81d44afaabc247fd1449767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.382397 24581 tablet_service.cc:1467] Processing CreateTablet for tablet bddb4a31d81d44afaabc247fd1449767 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=adc0aa06508a406b91ce8b5a5a44ea29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:13.382826 24581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bddb4a31d81d44afaabc247fd1449767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.385033 24846 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce01a3c9d9948ba8f153ad150cf3cb0 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=adc0aa06508a406b91ce8b5a5a44ea29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:13.385234 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce01a3c9d9948ba8f153ad150cf3cb0 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=adc0aa06508a406b91ce8b5a5a44ea29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:13.385653 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce01a3c9d9948ba8f153ad150cf3cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.386044 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce01a3c9d9948ba8f153ad150cf3cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.387213 24848 tablet_service.cc:1467] Processing CreateTablet for tablet 42957eb0c3864381b67438035ec91c28 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=adc0aa06508a406b91ce8b5a5a44ea29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:13.393136 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42957eb0c3864381b67438035ec91c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.395180 26540 tablet_bootstrap.cc:492] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:13.397915 26540 tablet_bootstrap.cc:654] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.398177 26543 tablet_bootstrap.cc:492] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:13.400391 26541 tablet_bootstrap.cc:492] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:13.402138 26543 tablet_bootstrap.cc:654] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.404631 26545 tablet_bootstrap.cc:492] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:13.405232 24845 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:13.406250 26542 tablet_bootstrap.cc:492] T 42957eb0c3864381b67438035ec91c28 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:13.414121 26545 tablet_bootstrap.cc:654] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.415663 26541 tablet_bootstrap.cc:654] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.425081 26542 tablet_bootstrap.cc:654] T 42957eb0c3864381b67438035ec91c28 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.430368 26543 tablet_bootstrap.cc:492] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:13.430675 26543 ts_tablet_manager.cc:1367] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.004s
I20240502 01:54:13.431648 26543 raft_consensus.cc:348] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:13.431802 21709 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:13.432030 26543 raft_consensus.cc:374] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.432255 26543 raft_consensus.cc:729] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.432588 26543 consensus_queue.cc:260] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:13.443140 26524 tablet.cc:1722] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:13.448719 26531 tablet.cc:1722] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:13.455267 26525 tablet.cc:1722] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:13.463698 26542 tablet_bootstrap.cc:492] T 42957eb0c3864381b67438035ec91c28 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:13.463857 21710 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=6ea7ec30ec06464fa77ecc96efb15aaf] alter complete (version 3)
I20240502 01:54:13.465027 26541 tablet_bootstrap.cc:492] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:13.465256 26541 ts_tablet_manager.cc:1367] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.003s
I20240502 01:54:13.465473 26542 ts_tablet_manager.cc:1367] T 42957eb0c3864381b67438035ec91c28 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.003s
I20240502 01:54:13.466430 26541 raft_consensus.cc:348] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.467131 26543 ts_tablet_manager.cc:1398] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20240502 01:54:13.468190 26542 raft_consensus.cc:348] T 42957eb0c3864381b67438035ec91c28 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.474357 26543 tablet_bootstrap.cc:492] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:13.474999 26540 tablet_bootstrap.cc:492] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:13.475183 26542 raft_consensus.cc:374] T 42957eb0c3864381b67438035ec91c28 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.475381 26541 raft_consensus.cc:374] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.475550 26542 raft_consensus.cc:729] T 42957eb0c3864381b67438035ec91c28 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.475729 26541 raft_consensus.cc:729] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.476087 26542 consensus_queue.cc:260] T 42957eb0c3864381b67438035ec91c28 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.476567 26541 consensus_queue.cc:260] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.475165 26540 ts_tablet_manager.cc:1367] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.080s user 0.013s sys 0.023s
I20240502 01:54:13.477852 26541 ts_tablet_manager.cc:1398] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240502 01:54:13.478649 26542 ts_tablet_manager.cc:1398] T 42957eb0c3864381b67438035ec91c28 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240502 01:54:13.480021 26543 tablet_bootstrap.cc:654] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.486155 26545 tablet_bootstrap.cc:492] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:13.486407 26545 ts_tablet_manager.cc:1367] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.082s user 0.006s sys 0.000s
I20240502 01:54:13.487504 26545 raft_consensus.cc:348] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:13.487861 26545 raft_consensus.cc:374] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.488006 26545 raft_consensus.cc:729] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.488334 26545 consensus_queue.cc:260] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:13.489262 26545 ts_tablet_manager.cc:1398] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:13.489902 26545 tablet_bootstrap.cc:492] T bddb4a31d81d44afaabc247fd1449767 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:13.492667 26545 tablet_bootstrap.cc:654] T bddb4a31d81d44afaabc247fd1449767 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.494065 26540 raft_consensus.cc:348] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:13.494334 26540 raft_consensus.cc:374] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.494441 26540 raft_consensus.cc:729] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.494819 26540 consensus_queue.cc:260] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:13.499142 26540 ts_tablet_manager.cc:1398] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20240502 01:54:13.504951 26545 tablet_bootstrap.cc:492] T bddb4a31d81d44afaabc247fd1449767 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:13.505705 26545 ts_tablet_manager.cc:1367] T bddb4a31d81d44afaabc247fd1449767 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20240502 01:54:13.509447 26545 raft_consensus.cc:348] T bddb4a31d81d44afaabc247fd1449767 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.509698 26545 raft_consensus.cc:374] T bddb4a31d81d44afaabc247fd1449767 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.509788 26545 raft_consensus.cc:729] T bddb4a31d81d44afaabc247fd1449767 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.510198 26545 consensus_queue.cc:260] T bddb4a31d81d44afaabc247fd1449767 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.511042 26543 tablet_bootstrap.cc:492] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:13.511284 26543 ts_tablet_manager.cc:1367] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.002s
I20240502 01:54:13.512012 26545 ts_tablet_manager.cc:1398] T bddb4a31d81d44afaabc247fd1449767 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:54:13.513298 26545 tablet_bootstrap.cc:492] T 42957eb0c3864381b67438035ec91c28 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:13.517876 26545 tablet_bootstrap.cc:654] T 42957eb0c3864381b67438035ec91c28 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.521960 26543 raft_consensus.cc:348] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.522447 26543 raft_consensus.cc:374] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.522610 26543 raft_consensus.cc:729] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.523078 26543 consensus_queue.cc:260] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.528483 26543 ts_tablet_manager.cc:1398] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240502 01:54:13.533496 26543 tablet_bootstrap.cc:492] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:13.536443 26543 tablet_bootstrap.cc:654] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.541131 26545 tablet_bootstrap.cc:492] T 42957eb0c3864381b67438035ec91c28 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:13.541491 26545 ts_tablet_manager.cc:1367] T 42957eb0c3864381b67438035ec91c28 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.029s user 0.018s sys 0.006s
I20240502 01:54:13.542827 26545 raft_consensus.cc:348] T 42957eb0c3864381b67438035ec91c28 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.543262 26545 raft_consensus.cc:374] T 42957eb0c3864381b67438035ec91c28 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.543440 26545 raft_consensus.cc:729] T 42957eb0c3864381b67438035ec91c28 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.543855 26545 consensus_queue.cc:260] T 42957eb0c3864381b67438035ec91c28 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.546682 26545 ts_tablet_manager.cc:1398] T 42957eb0c3864381b67438035ec91c28 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:13.552069 26543 tablet_bootstrap.cc:492] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:13.552287 26543 ts_tablet_manager.cc:1367] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.013s
I20240502 01:54:13.553265 26543 raft_consensus.cc:348] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.553532 26543 raft_consensus.cc:374] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.553694 26543 raft_consensus.cc:729] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.553968 26543 consensus_queue.cc:260] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.554690 26543 ts_tablet_manager.cc:1398] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:13.559399 26462 raft_consensus.cc:482] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:13.559401 26322 raft_consensus.cc:482] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:13.559561 26462 raft_consensus.cc:504] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:13.559590 26322 raft_consensus.cc:504] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:13.560199 26462 leader_election.cc:288] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:13.560315 26322 leader_election.cc:288] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:13.560711 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce01a3c9d9948ba8f153ad150cf3cb0" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:13.560874 24867 raft_consensus.cc:2454] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:13.561189 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce01a3c9d9948ba8f153ad150cf3cb0" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:13.561303 24537 leader_election.cc:302] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:13.561457 22249 raft_consensus.cc:2454] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:13.561573 26197 raft_consensus.cc:482] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:13.561507 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce01a3c9d9948ba8f153ad150cf3cb0" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:13.561780 24602 raft_consensus.cc:2454] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:13.561755 26197 raft_consensus.cc:504] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.561869 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce01a3c9d9948ba8f153ad150cf3cb0" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:13.562141 22250 raft_consensus.cc:2454] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:13.562196 24802 leader_election.cc:302] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:13.562397 26197 leader_election.cc:288] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:13.562567 26197 raft_consensus.cc:2781] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:13.562670 26197 raft_consensus.cc:482] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:13.562775 26197 raft_consensus.cc:3037] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.562881 26322 raft_consensus.cc:2781] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:13.562959 26322 raft_consensus.cc:482] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:13.563071 26322 raft_consensus.cc:3037] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.565568 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42957eb0c3864381b67438035ec91c28" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:13.565824 25000 raft_consensus.cc:2454] T 42957eb0c3864381b67438035ec91c28 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:13.566076 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42957eb0c3864381b67438035ec91c28" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:13.566298 24868 raft_consensus.cc:2454] T 42957eb0c3864381b67438035ec91c28 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:13.566354 24539 leader_election.cc:302] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:13.566766 26462 raft_consensus.cc:2781] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:13.566884 26462 raft_consensus.cc:482] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:13.566962 26462 raft_consensus.cc:3037] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.567421 26322 raft_consensus.cc:504] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:13.568555 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce01a3c9d9948ba8f153ad150cf3cb0" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:13.568558 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce01a3c9d9948ba8f153ad150cf3cb0" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:13.568760 22250 raft_consensus.cc:3037] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.569283 26462 raft_consensus.cc:504] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.570206 26197 raft_consensus.cc:504] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:13.571051 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42957eb0c3864381b67438035ec91c28" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:13.571309 25000 raft_consensus.cc:3037] T 42957eb0c3864381b67438035ec91c28 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.572161 22250 raft_consensus.cc:2454] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:13.573928 25000 raft_consensus.cc:2454] T 42957eb0c3864381b67438035ec91c28 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:13.574280 21709 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:13.574553 21709 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:13.574990 24803 leader_election.cc:302] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:13.577221 24602 raft_consensus.cc:2379] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d7449bb1df214bf4a5f93e4631312f08 in current term 1: Already voted for candidate 0c64d40b5d7644bd8b29feaa80eab72f in this term.
I20240502 01:54:13.575937 24539 leader_election.cc:302] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:13.577884 26555 raft_consensus.cc:2781] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:13.578018 26462 leader_election.cc:288] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:13.578781 26197 leader_election.cc:288] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:13.579293 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce01a3c9d9948ba8f153ad150cf3cb0" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:13.579722 22250 raft_consensus.cc:2379] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0c64d40b5d7644bd8b29feaa80eab72f in current term 1: Already voted for candidate d7449bb1df214bf4a5f93e4631312f08 in this term.
I20240502 01:54:13.579968 26555 raft_consensus.cc:686] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:54:13.580327 26555 consensus_queue.cc:237] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.581430 21709 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet b3f8e9fb62f44e4d93a5183f5c137ef0
I20240502 01:54:13.582929 24715 tablet_service.cc:1514] Processing DeleteTablet for tablet b3f8e9fb62f44e4d93a5183f5c137ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:13 UTC) from {username='slave'} at 127.0.0.1:41650
I20240502 01:54:13.579685 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42957eb0c3864381b67438035ec91c28" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:13.583307 24868 raft_consensus.cc:3037] T 42957eb0c3864381b67438035ec91c28 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.583766 24581 tablet_service.cc:1514] Processing DeleteTablet for tablet b3f8e9fb62f44e4d93a5183f5c137ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:13 UTC) from {username='slave'} at 127.0.0.1:53832
I20240502 01:54:13.575413 26537 raft_consensus.cc:2781] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:13.581118 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce01a3c9d9948ba8f153ad150cf3cb0" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:13.586761 24979 tablet_service.cc:1514] Processing DeleteTablet for tablet b3f8e9fb62f44e4d93a5183f5c137ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:13 UTC) from {username='slave'} at 127.0.0.1:39964
I20240502 01:54:13.587975 24868 raft_consensus.cc:2454] T 42957eb0c3864381b67438035ec91c28 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:13.575306 26322 leader_election.cc:288] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:13.594285 26558 tablet_replica.cc:330] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:54:13.594818 26558 raft_consensus.cc:2229] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:13.595451 26558 raft_consensus.cc:2258] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:13.596174 26537 raft_consensus.cc:686] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:13.596936 26558 ts_tablet_manager.cc:1875] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:13.596741 26537 consensus_queue.cc:237] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:13.602003 26561 tablet_replica.cc:330] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:54:13.602684 26559 tablet_replica.cc:330] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:54:13.603056 26561 raft_consensus.cc:2229] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:13.603366 26559 raft_consensus.cc:2229] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:13.604130 26558 ts_tablet_manager.cc:1888] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:13.604192 26559 raft_consensus.cc:2258] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:13.603873 26561 raft_consensus.cc:2258] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:13.605509 26561 ts_tablet_manager.cc:1875] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:13.606024 26558 log.cc:1196] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/b3f8e9fb62f44e4d93a5183f5c137ef0
I20240502 01:54:13.606468 26559 ts_tablet_manager.cc:1875] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:13.607959 26558 ts_tablet_manager.cc:1909] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 28157bc3e678428fb5f93c077d528513: Deleting consensus metadata
I20240502 01:54:13.608244 21709 catalog_manager.cc:5484] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:13.609844 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet b3f8e9fb62f44e4d93a5183f5c137ef0 (table rebalancer_dynamic_table_new_name [id=6ea7ec30ec06464fa77ecc96efb15aaf]) successfully deleted
I20240502 01:54:13.610033 21715 catalog_manager.cc:5484] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:13.608933 24537 leader_election.cc:302] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: c2a28baa7ee2469294d94b7fe5455cff, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:13.614821 26555 raft_consensus.cc:2726] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:54:13.624835 26559 ts_tablet_manager.cc:1888] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:13.625042 26559 log.cc:1196] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/b3f8e9fb62f44e4d93a5183f5c137ef0
I20240502 01:54:13.625567 26559 ts_tablet_manager.cc:1909] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting consensus metadata
I20240502 01:54:13.626042 26561 ts_tablet_manager.cc:1888] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:13.626255 26561 log.cc:1196] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/b3f8e9fb62f44e4d93a5183f5c137ef0
I20240502 01:54:13.626847 26561 ts_tablet_manager.cc:1909] T b3f8e9fb62f44e4d93a5183f5c137ef0 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting consensus metadata
I20240502 01:54:13.626945 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet b3f8e9fb62f44e4d93a5183f5c137ef0 (table rebalancer_dynamic_table_new_name [id=6ea7ec30ec06464fa77ecc96efb15aaf]) successfully deleted
I20240502 01:54:13.628032 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet b3f8e9fb62f44e4d93a5183f5c137ef0 (table rebalancer_dynamic_table_new_name [id=6ea7ec30ec06464fa77ecc96efb15aaf]) successfully deleted
I20240502 01:54:13.650068 21989 consensus_queue.cc:237] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:13.652082 21857 raft_consensus.cc:1261] T ec67f45b2b604c129d8572c5ca6acbc4 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:13.652135 24602 raft_consensus.cc:1261] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:13.652729 26429 consensus_queue.cc:1035] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:13.653093 26539 consensus_queue.cc:1035] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:13.655961 26539 raft_consensus.cc:2932] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:13.656919 24602 raft_consensus.cc:2932] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:13.657101 21857 raft_consensus.cc:2932] T ec67f45b2b604c129d8572c5ca6acbc4 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:13.662307 21715 catalog_manager.cc:5484] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:13.667961 26539 raft_consensus.cc:482] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:13.668277 26539 raft_consensus.cc:504] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.669800 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet ec67f45b2b604c129d8572c5ca6acbc4 with delete_type TABLET_DATA_TOMBSTONED (TS c64840657cdf4b93980bd34120c94d9a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:13.670241 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddb4a31d81d44afaabc247fd1449767" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:13.670539 24867 raft_consensus.cc:2454] T bddb4a31d81d44afaabc247fd1449767 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:13.671574 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddb4a31d81d44afaabc247fd1449767" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:13.671850 24602 raft_consensus.cc:2454] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:13.671918 26584 tablet_replica.cc:330] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:13.672230 26584 raft_consensus.cc:2229] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:13.672605 26584 raft_consensus.cc:2258] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:13.673449 26584 ts_tablet_manager.cc:1875] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:13.674664 21923 leader_election.cc:302] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:13.675096 26429 raft_consensus.cc:2781] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:13.675185 26539 leader_election.cc:288] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:13.675246 26429 raft_consensus.cc:482] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:13.675350 26429 raft_consensus.cc:3037] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.678367 26429 raft_consensus.cc:504] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.679571 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddb4a31d81d44afaabc247fd1449767" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:13.679832 24602 raft_consensus.cc:3037] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.682682 24602 raft_consensus.cc:2454] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:13.683290 21923 leader_election.cc:302] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 0c64d40b5d7644bd8b29feaa80eab72f; no voters:
I20240502 01:54:13.684309 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddb4a31d81d44afaabc247fd1449767" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:13.684587 24867 raft_consensus.cc:3037] T bddb4a31d81d44afaabc247fd1449767 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.685987 26584 ts_tablet_manager.cc:1888] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:54:13.686141 26584 log.cc:1196] T ec67f45b2b604c129d8572c5ca6acbc4 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/ec67f45b2b604c129d8572c5ca6acbc4
I20240502 01:54:13.686606 21709 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:13.687391 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet ec67f45b2b604c129d8572c5ca6acbc4 (table rebalance_test_table_7 [id=cd297aa2796f428db6b815990620e2a4]) successfully deleted
I20240502 01:54:13.687979 24867 raft_consensus.cc:2454] T bddb4a31d81d44afaabc247fd1449767 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:13.688283 26429 leader_election.cc:288] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:13.688542 26429 raft_consensus.cc:2781] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:13.688726 26429 raft_consensus.cc:686] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:54:13.689091 26429 consensus_queue.cc:237] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.696794 21709 catalog_manager.cc:5484] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:13.697614 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbcd6dbe47d4922a6861d067be57054 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f3ec7fdc982b4e81857258f95ab8827c]), partition=
I20240502 01:54:13.698046 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbcd6dbe47d4922a6861d067be57054. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.703956 24846 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbcd6dbe47d4922a6861d067be57054 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f3ec7fdc982b4e81857258f95ab8827c]), partition=
I20240502 01:54:13.703956 24979 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbcd6dbe47d4922a6861d067be57054 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f3ec7fdc982b4e81857258f95ab8827c]), partition=
I20240502 01:54:13.704371 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbcd6dbe47d4922a6861d067be57054. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.704371 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbcd6dbe47d4922a6861d067be57054. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.712232 26542 tablet_bootstrap.cc:492] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:13.713976 26542 tablet_bootstrap.cc:654] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.714174 26545 tablet_bootstrap.cc:492] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:13.715984 26545 tablet_bootstrap.cc:654] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.716609 26542 tablet_bootstrap.cc:492] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:13.716781 26542 ts_tablet_manager.cc:1367] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 01:54:13.717741 26542 raft_consensus.cc:348] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.718127 26542 raft_consensus.cc:374] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.718225 26542 raft_consensus.cc:729] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.718538 26542 consensus_queue.cc:260] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.719588 26542 ts_tablet_manager.cc:1398] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:54:13.722025 26591 tablet_bootstrap.cc:492] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:13.722404 26545 tablet_bootstrap.cc:492] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:13.722671 26545 ts_tablet_manager.cc:1367] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20240502 01:54:13.723482 26545 raft_consensus.cc:348] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.723773 26545 raft_consensus.cc:374] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.723908 26591 tablet_bootstrap.cc:654] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.723913 26545 raft_consensus.cc:729] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.725268 26545 consensus_queue.cc:260] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.726338 26545 ts_tablet_manager.cc:1398] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:13.731478 26591 tablet_bootstrap.cc:492] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:13.731691 26591 ts_tablet_manager.cc:1367] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240502 01:54:13.732779 26591 raft_consensus.cc:348] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.733132 26591 raft_consensus.cc:374] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.733258 26591 raft_consensus.cc:729] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.733566 26591 consensus_queue.cc:260] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:13.734537 26591 ts_tablet_manager.cc:1398] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:54:13.959151 26555 consensus_queue.cc:1035] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.980336 26462 consensus_queue.cc:1035] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:14.003924 26473 raft_consensus.cc:482] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:14.004130 26473 raft_consensus.cc:504] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:14.004803 26473 leader_election.cc:288] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:14.005221 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbcd6dbe47d4922a6861d067be57054" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:14.005468 25000 raft_consensus.cc:2454] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:14.005429 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbcd6dbe47d4922a6861d067be57054" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:14.005770 24867 raft_consensus.cc:2454] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:14.006058 24671 leader_election.cc:302] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:14.006371 26473 raft_consensus.cc:2781] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:14.006538 26473 raft_consensus.cc:482] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:14.006631 26473 raft_consensus.cc:3037] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.009029 26473 raft_consensus.cc:504] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:14.009613 26473 leader_election.cc:288] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:14.010131 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbcd6dbe47d4922a6861d067be57054" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:14.010195 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbcd6dbe47d4922a6861d067be57054" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:14.010375 25000 raft_consensus.cc:3037] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.010428 24867 raft_consensus.cc:3037] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.013121 24867 raft_consensus.cc:2454] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:14.013176 25000 raft_consensus.cc:2454] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:14.013628 24669 leader_election.cc:302] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:14.013995 26041 raft_consensus.cc:2781] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:14.014250 26041 raft_consensus.cc:686] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:14.014562 26041 consensus_queue.cc:237] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:14.018277 21715 catalog_manager.cc:5484] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:14.064837 21967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:14.090493 26537 consensus_queue.cc:1035] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:14.100455 24979 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:14.110687 22100 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:14.129509 26560 consensus_queue.cc:1035] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:14.130954 24715 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:54:14.140434 22053 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.193:33785, user_credentials={real_user=slave}} blocked reactor thread for 64202.9us
I20240502 01:54:14.142079 22231 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:14.153788 24581 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:14.175858 21836 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:14.179533 24846 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:14.180415 21715 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:14.209265 26429 consensus_queue.cc:1035] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:14.214222 24868 raft_consensus.cc:1261] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:14.215178 26041 consensus_queue.cc:1035] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:14.235689 26539 consensus_queue.cc:1035] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:14.236866 25000 raft_consensus.cc:1261] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:14.237623 26041 consensus_queue.cc:1035] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:14.246220 21710 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:14.250669 21715 catalog_manager.cc:5576] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737) does not have the latest schema for tablet 9dbcd6dbe47d4922a6861d067be57054 (table rebalancer_dynamic_table [id=f3ec7fdc982b4e81857258f95ab8827c]). Expected version 1 got 0
I20240502 01:54:14.262913 21709 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 9dbcd6dbe47d4922a6861d067be57054 (table rebalancer_dynamic_table [id=f3ec7fdc982b4e81857258f95ab8827c]). Expected version 1 got 0
I20240502 01:54:14.269580 24979 tablet_service.cc:1467] Processing CreateTablet for tablet 0bada775411248a3a7fbbeca2a9eea0c (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=53be2be99d0d46df8778de508abff3a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.270133 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bada775411248a3a7fbbeca2a9eea0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.270677 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 3237002dce97460799e219cf5b4968cb (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=53be2be99d0d46df8778de508abff3a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.270841 24846 tablet_service.cc:1467] Processing CreateTablet for tablet 3237002dce97460799e219cf5b4968cb (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=53be2be99d0d46df8778de508abff3a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.271119 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3237002dce97460799e219cf5b4968cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.271260 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3237002dce97460799e219cf5b4968cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.272919 24713 tablet_service.cc:1467] Processing CreateTablet for tablet 0bada775411248a3a7fbbeca2a9eea0c (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=53be2be99d0d46df8778de508abff3a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.273221 24714 tablet_service.cc:1467] Processing CreateTablet for tablet 00c38c0987ed469eba1fb6f76ee2f6a9 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=53be2be99d0d46df8778de508abff3a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.273334 24713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bada775411248a3a7fbbeca2a9eea0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.273561 24714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c38c0987ed469eba1fb6f76ee2f6a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.273833 24581 tablet_service.cc:1467] Processing CreateTablet for tablet 3237002dce97460799e219cf5b4968cb (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=53be2be99d0d46df8778de508abff3a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.274296 24581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3237002dce97460799e219cf5b4968cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.277262 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 0bada775411248a3a7fbbeca2a9eea0c (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=53be2be99d0d46df8778de508abff3a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.277738 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bada775411248a3a7fbbeca2a9eea0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.281327 24582 tablet_service.cc:1467] Processing CreateTablet for tablet 00c38c0987ed469eba1fb6f76ee2f6a9 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=53be2be99d0d46df8778de508abff3a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.281736 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c38c0987ed469eba1fb6f76ee2f6a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.285347 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 00c38c0987ed469eba1fb6f76ee2f6a9 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=53be2be99d0d46df8778de508abff3a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.285969 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c38c0987ed469eba1fb6f76ee2f6a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.288472 26661 tablet_bootstrap.cc:492] T 0bada775411248a3a7fbbeca2a9eea0c P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:14.299345 26660 tablet.cc:1722] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:14.310257 26658 tablet.cc:1722] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:14.310902 26658 tablet.cc:1620] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:14.313799 26661 tablet_bootstrap.cc:654] T 0bada775411248a3a7fbbeca2a9eea0c P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.321175 21715 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 9dbcd6dbe47d4922a6861d067be57054 (table rebalancer_dynamic_table [id=f3ec7fdc982b4e81857258f95ab8827c]). Expected version 1 got 0
I20240502 01:54:14.323212 26661 tablet_bootstrap.cc:492] T 0bada775411248a3a7fbbeca2a9eea0c P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:14.323386 26661 ts_tablet_manager.cc:1367] T 0bada775411248a3a7fbbeca2a9eea0c P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20240502 01:54:14.326541 26667 tablet_bootstrap.cc:492] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:14.326767 26661 raft_consensus.cc:348] T 0bada775411248a3a7fbbeca2a9eea0c P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:14.326997 26661 raft_consensus.cc:374] T 0bada775411248a3a7fbbeca2a9eea0c P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.327095 26661 raft_consensus.cc:729] T 0bada775411248a3a7fbbeca2a9eea0c P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.327955 26661 consensus_queue.cc:260] T 0bada775411248a3a7fbbeca2a9eea0c P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:14.333608 26661 ts_tablet_manager.cc:1398] T 0bada775411248a3a7fbbeca2a9eea0c P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240502 01:54:14.339365 26667 tablet_bootstrap.cc:654] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.348377 26666 tablet_bootstrap.cc:492] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:14.351317 26668 tablet_bootstrap.cc:492] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:14.353614 26668 tablet_bootstrap.cc:654] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.355192 26659 tablet.cc:1722] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:14.355871 26659 tablet.cc:1620] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:14.361896 26660 tablet.cc:1620] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:14.362484 26666 tablet_bootstrap.cc:654] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.363842 21715 catalog_manager.cc:5576] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737) does not have the latest schema for tablet 9dbcd6dbe47d4922a6861d067be57054 (table rebalancer_dynamic_table [id=f3ec7fdc982b4e81857258f95ab8827c]). Expected version 1 got 0
I20240502 01:54:14.369611 26662 tablet_bootstrap.cc:492] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:14.378907 21715 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=f3ec7fdc982b4e81857258f95ab8827c] alter complete (version 1)
I20240502 01:54:14.379491 26668 tablet_bootstrap.cc:492] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:14.381171 26667 tablet_bootstrap.cc:492] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:14.381337 26667 ts_tablet_manager.cc:1367] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.000s
I20240502 01:54:14.381583 26668 ts_tablet_manager.cc:1367] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240502 01:54:14.382256 26667 raft_consensus.cc:348] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:14.382467 26667 raft_consensus.cc:374] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.382575 26667 raft_consensus.cc:729] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.385870 26667 consensus_queue.cc:260] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:14.384042 26668 raft_consensus.cc:348] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:14.389489 26668 raft_consensus.cc:374] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.389690 26668 raft_consensus.cc:729] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.390077 26668 consensus_queue.cc:260] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:14.392539 26662 tablet_bootstrap.cc:654] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.393702 26667 ts_tablet_manager.cc:1398] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240502 01:54:14.394035 26666 tablet_bootstrap.cc:492] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:14.394255 26666 ts_tablet_manager.cc:1367] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20240502 01:54:14.395144 26666 raft_consensus.cc:348] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:14.395411 26666 raft_consensus.cc:374] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.395534 26666 raft_consensus.cc:729] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.395855 26666 consensus_queue.cc:260] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:14.400270 26666 ts_tablet_manager.cc:1398] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:54:14.408272 26666 tablet_bootstrap.cc:492] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:14.410696 26668 ts_tablet_manager.cc:1398] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20240502 01:54:14.421213 26666 tablet_bootstrap.cc:654] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.423291 26662 tablet_bootstrap.cc:492] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:14.423467 26662 ts_tablet_manager.cc:1367] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.003s
I20240502 01:54:14.424270 26662 raft_consensus.cc:348] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:14.424486 26662 raft_consensus.cc:374] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.424587 26662 raft_consensus.cc:729] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.424873 26662 consensus_queue.cc:260] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:14.427006 26662 ts_tablet_manager.cc:1398] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:14.433552 26662 tablet_bootstrap.cc:492] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:14.442485 26662 tablet_bootstrap.cc:654] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.458730 26662 tablet_bootstrap.cc:492] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:14.458932 26662 ts_tablet_manager.cc:1367] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.001s
I20240502 01:54:14.459739 26662 raft_consensus.cc:348] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:14.459973 26662 raft_consensus.cc:374] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.460032 26662 raft_consensus.cc:729] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.460284 26662 consensus_queue.cc:260] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:14.461069 26662 ts_tablet_manager.cc:1398] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:54:14.465498 26666 tablet_bootstrap.cc:492] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:14.465691 26666 ts_tablet_manager.cc:1367] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.057s user 0.007s sys 0.005s
I20240502 01:54:14.466692 26666 raft_consensus.cc:348] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:14.466928 26666 raft_consensus.cc:374] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.467005 26666 raft_consensus.cc:729] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.467293 26666 consensus_queue.cc:260] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:14.467499 26662 tablet_bootstrap.cc:492] T 0bada775411248a3a7fbbeca2a9eea0c P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:14.467837 26666 ts_tablet_manager.cc:1398] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:14.468165 26666 tablet_bootstrap.cc:492] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:14.469731 26662 tablet_bootstrap.cc:654] T 0bada775411248a3a7fbbeca2a9eea0c P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.473044 26666 tablet_bootstrap.cc:654] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.480645 26666 tablet_bootstrap.cc:492] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:14.480811 26666 ts_tablet_manager.cc:1367] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20240502 01:54:14.481598 26666 raft_consensus.cc:348] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:14.482833 26666 raft_consensus.cc:374] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.482957 26666 raft_consensus.cc:729] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.483223 26666 consensus_queue.cc:260] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:14.483829 26666 ts_tablet_manager.cc:1398] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:14.485069 26662 tablet_bootstrap.cc:492] T 0bada775411248a3a7fbbeca2a9eea0c P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:14.485110 26560 raft_consensus.cc:482] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:14.485245 26662 ts_tablet_manager.cc:1367] T 0bada775411248a3a7fbbeca2a9eea0c P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.006s
I20240502 01:54:14.485309 26560 raft_consensus.cc:504] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:14.486143 26560 leader_election.cc:288] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:14.486263 26662 raft_consensus.cc:348] T 0bada775411248a3a7fbbeca2a9eea0c P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:14.486491 26662 raft_consensus.cc:374] T 0bada775411248a3a7fbbeca2a9eea0c P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.486634 26662 raft_consensus.cc:729] T 0bada775411248a3a7fbbeca2a9eea0c P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.486900 26662 consensus_queue.cc:260] T 0bada775411248a3a7fbbeca2a9eea0c P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:14.487073 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3237002dce97460799e219cf5b4968cb" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:14.487304 24603 raft_consensus.cc:2454] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:14.487651 26662 ts_tablet_manager.cc:1398] T 0bada775411248a3a7fbbeca2a9eea0c P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:14.489145 24802 leader_election.cc:302] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:14.491462 26560 raft_consensus.cc:2781] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:14.491602 26560 raft_consensus.cc:482] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:14.491763 26560 raft_consensus.cc:3037] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.495863 26560 raft_consensus.cc:504] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:14.497495 24733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3237002dce97460799e219cf5b4968cb" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:14.497818 24733 raft_consensus.cc:3037] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.497859 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3237002dce97460799e219cf5b4968cb" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:14.498101 24603 raft_consensus.cc:3037] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.505420 24603 raft_consensus.cc:2454] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:14.500861 24733 raft_consensus.cc:2454] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:14.501011 24734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3237002dce97460799e219cf5b4968cb" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:14.506660 24734 raft_consensus.cc:2362] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1. Re-sending same reply.
I20240502 01:54:14.509150 26560 leader_election.cc:288] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:14.509140 24802 leader_election.cc:302] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:14.509780 26560 raft_consensus.cc:2781] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:14.509964 26560 raft_consensus.cc:686] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:14.510250 26560 consensus_queue.cc:237] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:14.514711 21709 catalog_manager.cc:5484] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:14.524533 21715 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:14.534857 26658 tablet.cc:1722] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:14.535565 26658 tablet.cc:1620] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:14.546185 26659 tablet.cc:1722] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:14.546809 26659 tablet.cc:1620] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:14.569592 21715 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=f3ec7fdc982b4e81857258f95ab8827c] alter complete (version 2)
I20240502 01:54:14.571817 26555 raft_consensus.cc:482] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:14.572105 26555 raft_consensus.cc:504] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:14.573585 26555 leader_election.cc:288] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:14.574569 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bada775411248a3a7fbbeca2a9eea0c" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:14.574910 25001 raft_consensus.cc:2454] T 0bada775411248a3a7fbbeca2a9eea0c P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:14.575642 24539 leader_election.cc:302] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:14.575663 24734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bada775411248a3a7fbbeca2a9eea0c" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:14.575893 24734 raft_consensus.cc:2454] T 0bada775411248a3a7fbbeca2a9eea0c P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:14.576387 26660 tablet.cc:1722] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:14.576984 26660 tablet.cc:1620] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:14.577075 26555 raft_consensus.cc:2781] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:14.577214 26555 raft_consensus.cc:482] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:14.577363 26555 raft_consensus.cc:3037] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.580049 26555 raft_consensus.cc:504] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:14.581177 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bada775411248a3a7fbbeca2a9eea0c" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:14.581423 25001 raft_consensus.cc:3037] T 0bada775411248a3a7fbbeca2a9eea0c P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.581627 24734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bada775411248a3a7fbbeca2a9eea0c" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:14.581830 24734 raft_consensus.cc:3037] T 0bada775411248a3a7fbbeca2a9eea0c P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.584224 25001 raft_consensus.cc:2454] T 0bada775411248a3a7fbbeca2a9eea0c P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:14.584461 24734 raft_consensus.cc:2454] T 0bada775411248a3a7fbbeca2a9eea0c P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:14.584933 24537 leader_election.cc:302] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:54:14.585250 26555 leader_election.cc:288] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:14.585444 26555 raft_consensus.cc:2781] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:14.585575 26555 raft_consensus.cc:686] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:54:14.585877 26555 consensus_queue.cc:237] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:14.593557 21709 catalog_manager.cc:5484] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:14.614892 21988 consensus_queue.cc:237] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:14.617413 24866 raft_consensus.cc:1261] T db96ccbe81564bb589b0de65977b3321 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:14.617527 21857 raft_consensus.cc:1261] T db96ccbe81564bb589b0de65977b3321 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:14.618201 26429 consensus_queue.cc:1035] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:14.618512 26664 consensus_queue.cc:1035] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:14.621479 26539 raft_consensus.cc:2932] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:14.622305 21857 raft_consensus.cc:2932] T db96ccbe81564bb589b0de65977b3321 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:14.625715 24866 raft_consensus.cc:2932] T db96ccbe81564bb589b0de65977b3321 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:14.630491 21715 catalog_manager.cc:5484] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:14.637617 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet db96ccbe81564bb589b0de65977b3321 with delete_type TABLET_DATA_TOMBSTONED (TS c64840657cdf4b93980bd34120c94d9a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:14.647030 26473 raft_consensus.cc:482] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:14.647200 26473 raft_consensus.cc:504] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:14.647804 26473 leader_election.cc:288] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:14.648473 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c38c0987ed469eba1fb6f76ee2f6a9" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:54:14.648715 22118 raft_consensus.cc:2454] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:14.649034 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c38c0987ed469eba1fb6f76ee2f6a9" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:14.649235 24670 leader_election.cc:302] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:54:14.649269 24603 raft_consensus.cc:2454] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:14.650337 26473 raft_consensus.cc:2781] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:14.650462 26473 raft_consensus.cc:482] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:14.650563 26473 raft_consensus.cc:3037] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.652365 26703 tablet_replica.cc:330] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:14.652684 26473 raft_consensus.cc:504] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:14.653012 26703 raft_consensus.cc:2229] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:14.653512 26703 raft_consensus.cc:2258] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:14.653748 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c38c0987ed469eba1fb6f76ee2f6a9" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:14.653827 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c38c0987ed469eba1fb6f76ee2f6a9" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:54:14.653977 24603 raft_consensus.cc:3037] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.654031 22118 raft_consensus.cc:3037] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.654944 26703 ts_tablet_manager.cc:1875] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:14.656955 26473 leader_election.cc:288] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:14.657325 24603 raft_consensus.cc:2454] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:14.657397 22118 raft_consensus.cc:2454] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:14.657876 24670 leader_election.cc:302] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:54:14.658222 26473 raft_consensus.cc:2781] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:14.658442 26473 raft_consensus.cc:686] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:14.658754 26473 consensus_queue.cc:237] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:14.662482 21713 catalog_manager.cc:5484] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:14.664076 26703 ts_tablet_manager.cc:1888] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:54:14.664270 26703 log.cc:1196] T db96ccbe81564bb589b0de65977b3321 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/db96ccbe81564bb589b0de65977b3321
I20240502 01:54:14.665287 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet db96ccbe81564bb589b0de65977b3321 (table rebalance_test_table_3 [id=27064ffcfcc643d6ad082b6cd6ea21db]) successfully deleted
I20240502 01:54:14.679106 21713 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:14.687460 26658 tablet.cc:1722] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:14.688727 26659 tablet.cc:1722] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:14.688750 26660 tablet.cc:1722] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:14.701346 21713 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=f3ec7fdc982b4e81857258f95ab8827c] alter complete (version 3)
I20240502 01:54:14.802089 21715 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:14.802404 21715 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:14.806056 21715 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 9dbcd6dbe47d4922a6861d067be57054
I20240502 01:54:14.808243 24846 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dbcd6dbe47d4922a6861d067be57054 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:14 UTC) from {username='slave'} at 127.0.0.1:51592
I20240502 01:54:14.808766 24715 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dbcd6dbe47d4922a6861d067be57054 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:14 UTC) from {username='slave'} at 127.0.0.1:41650
I20240502 01:54:14.809149 26746 tablet_replica.cc:330] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08: stopping tablet replica
I20240502 01:54:14.809419 26746 raft_consensus.cc:2229] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:14.809757 26746 raft_consensus.cc:2258] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:14.816742 24979 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dbcd6dbe47d4922a6861d067be57054 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:14 UTC) from {username='slave'} at 127.0.0.1:39964
I20240502 01:54:14.836653 26746 ts_tablet_manager.cc:1875] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:14.851178 26748 tablet_replica.cc:330] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:54:14.851666 26747 tablet_replica.cc:330] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:54:14.851954 26747 raft_consensus.cc:2229] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:14.852538 26748 raft_consensus.cc:2229] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:14.853044 26747 raft_consensus.cc:2258] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:14.853343 26748 raft_consensus.cc:2258] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:14.854012 26746 ts_tablet_manager.cc:1888] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:14.854233 26746 log.cc:1196] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/wals/9dbcd6dbe47d4922a6861d067be57054
I20240502 01:54:14.854548 26748 ts_tablet_manager.cc:1875] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:14.854862 26746 ts_tablet_manager.cc:1909] T 9dbcd6dbe47d4922a6861d067be57054 P d7449bb1df214bf4a5f93e4631312f08: Deleting consensus metadata
I20240502 01:54:14.855405 26747 ts_tablet_manager.cc:1875] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:14.856114 21700 catalog_manager.cc:4830] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): tablet 9dbcd6dbe47d4922a6861d067be57054 (table rebalancer_dynamic_table_new_name [id=f3ec7fdc982b4e81857258f95ab8827c]) successfully deleted
I20240502 01:54:14.873591 26748 ts_tablet_manager.cc:1888] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:14.873764 26748 log.cc:1196] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/9dbcd6dbe47d4922a6861d067be57054
I20240502 01:54:14.878371 26747 ts_tablet_manager.cc:1888] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:14.878562 26747 log.cc:1196] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/9dbcd6dbe47d4922a6861d067be57054
I20240502 01:54:14.874274 26748 ts_tablet_manager.cc:1909] T 9dbcd6dbe47d4922a6861d067be57054 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting consensus metadata
I20240502 01:54:14.880613 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet 9dbcd6dbe47d4922a6861d067be57054 (table rebalancer_dynamic_table_new_name [id=f3ec7fdc982b4e81857258f95ab8827c]) successfully deleted
I20240502 01:54:14.881733 26747 ts_tablet_manager.cc:1909] T 9dbcd6dbe47d4922a6861d067be57054 P 28157bc3e678428fb5f93c077d528513: Deleting consensus metadata
I20240502 01:54:14.883654 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet 9dbcd6dbe47d4922a6861d067be57054 (table rebalancer_dynamic_table_new_name [id=f3ec7fdc982b4e81857258f95ab8827c]) successfully deleted
I20240502 01:54:14.909735 21713 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:14.921263 24846 tablet_service.cc:1467] Processing CreateTablet for tablet 904a7757410c41f4af5b56ea661f2371 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2cd4d985b71d4cf0a0a6fc503d57c436]), partition=
I20240502 01:54:14.921599 21967 tablet_service.cc:1467] Processing CreateTablet for tablet 904a7757410c41f4af5b56ea661f2371 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2cd4d985b71d4cf0a0a6fc503d57c436]), partition=
I20240502 01:54:14.921711 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904a7757410c41f4af5b56ea661f2371. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.922082 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904a7757410c41f4af5b56ea661f2371. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.927605 24979 tablet_service.cc:1467] Processing CreateTablet for tablet 904a7757410c41f4af5b56ea661f2371 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2cd4d985b71d4cf0a0a6fc503d57c436]), partition=
I20240502 01:54:14.928040 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904a7757410c41f4af5b56ea661f2371. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.930594 21969 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:14.955644 26765 tablet_bootstrap.cc:492] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:14.962971 26764 tablet_bootstrap.cc:492] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:14.966179 26764 tablet_bootstrap.cc:654] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.971204 26555 consensus_queue.cc:1035] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:14.977782 24981 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:14.989759 26765 tablet_bootstrap.cc:654] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.993772 21836 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:14.994197 26766 tablet_bootstrap.cc:492] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:14.996505 26766 tablet_bootstrap.cc:654] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.010375 26764 tablet_bootstrap.cc:492] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:15.010603 26764 ts_tablet_manager.cc:1367] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.030s
I20240502 01:54:15.011693 26764 raft_consensus.cc:348] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:15.012027 26764 raft_consensus.cc:374] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.012149 26764 raft_consensus.cc:729] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.012380 26764 consensus_queue.cc:260] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:15.013088 26764 ts_tablet_manager.cc:1398] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20240502 01:54:15.013746 26462 consensus_queue.cc:1035] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.030440 26560 consensus_queue.cc:1035] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.054347 26765 tablet_bootstrap.cc:492] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:15.054628 26765 ts_tablet_manager.cc:1367] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.099s user 0.018s sys 0.028s
I20240502 01:54:15.055701 26765 raft_consensus.cc:348] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:15.056038 26765 raft_consensus.cc:374] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.056208 26765 raft_consensus.cc:729] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.056393 26766 tablet_bootstrap.cc:492] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:15.056576 26766 ts_tablet_manager.cc:1367] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.002s
I20240502 01:54:15.056579 26765 consensus_queue.cc:260] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:15.057387 26765 ts_tablet_manager.cc:1398] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:15.057989 22100 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:15.058171 26766 raft_consensus.cc:348] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:15.058591 26766 raft_consensus.cc:374] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.058697 26766 raft_consensus.cc:729] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.058951 26766 consensus_queue.cc:260] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:15.059461 26766 ts_tablet_manager.cc:1398] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:54:15.063385 22231 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:15.070228 24583 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:15.079564 24846 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:15.084082 26560 consensus_queue.cc:1035] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.127060 24715 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:15.159021 26560 raft_consensus.cc:482] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:15.159225 26560 raft_consensus.cc:504] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:15.159933 26560 leader_election.cc:288] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:15.166445 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904a7757410c41f4af5b56ea661f2371" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:15.166735 25000 raft_consensus.cc:2454] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:15.168238 24804 leader_election.cc:302] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:15.178081 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904a7757410c41f4af5b56ea661f2371" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:54:15.178376 21988 raft_consensus.cc:2454] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:15.180629 26560 raft_consensus.cc:2781] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:15.180794 26560 raft_consensus.cc:482] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:15.180938 26560 raft_consensus.cc:3037] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.183907 26560 raft_consensus.cc:504] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:15.184548 26560 leader_election.cc:288] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:15.185130 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904a7757410c41f4af5b56ea661f2371" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:15.185158 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904a7757410c41f4af5b56ea661f2371" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:54:15.185364 25000 raft_consensus.cc:3037] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.185447 21988 raft_consensus.cc:3037] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.192238 21988 raft_consensus.cc:2454] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:15.192718 24801 leader_election.cc:302] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:15.193430 25000 raft_consensus.cc:2454] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:15.193786 26560 raft_consensus.cc:2781] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:15.193953 26560 raft_consensus.cc:686] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:15.194270 26560 consensus_queue.cc:237] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:15.208091 21712 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:15.209059 21713 catalog_manager.cc:5484] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:15.230821 21967 tablet_service.cc:1467] Processing CreateTablet for tablet a9fb0b24337a4175972db268aac5df34 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=bb7acab87c2b4608b70144063a8a4a6e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.231429 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fb0b24337a4175972db268aac5df34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.231587 21969 tablet_service.cc:1467] Processing CreateTablet for tablet fafa4ef17fac41b68e3bf4aaa8a4caa8 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=bb7acab87c2b4608b70144063a8a4a6e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.231930 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafa4ef17fac41b68e3bf4aaa8a4caa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.232252 24979 tablet_service.cc:1467] Processing CreateTablet for tablet fafa4ef17fac41b68e3bf4aaa8a4caa8 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=bb7acab87c2b4608b70144063a8a4a6e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.232710 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafa4ef17fac41b68e3bf4aaa8a4caa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.236533 24981 tablet_service.cc:1467] Processing CreateTablet for tablet 6e8d8fbd2d1249d5b5de5f189b0d03e9 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=bb7acab87c2b4608b70144063a8a4a6e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.236719 24583 tablet_service.cc:1467] Processing CreateTablet for tablet a9fb0b24337a4175972db268aac5df34 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=bb7acab87c2b4608b70144063a8a4a6e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.236800 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 6e8d8fbd2d1249d5b5de5f189b0d03e9 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=bb7acab87c2b4608b70144063a8a4a6e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.236987 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e8d8fbd2d1249d5b5de5f189b0d03e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.237176 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fb0b24337a4175972db268aac5df34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.237303 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e8d8fbd2d1249d5b5de5f189b0d03e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.237270 24715 tablet_service.cc:1467] Processing CreateTablet for tablet a9fb0b24337a4175972db268aac5df34 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=bb7acab87c2b4608b70144063a8a4a6e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.237689 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fb0b24337a4175972db268aac5df34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.239893 24713 tablet_service.cc:1467] Processing CreateTablet for tablet fafa4ef17fac41b68e3bf4aaa8a4caa8 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=bb7acab87c2b4608b70144063a8a4a6e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.240314 24713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafa4ef17fac41b68e3bf4aaa8a4caa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.243911 26041 consensus_queue.cc:1035] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.245302 24714 tablet_service.cc:1467] Processing CreateTablet for tablet 6e8d8fbd2d1249d5b5de5f189b0d03e9 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=bb7acab87c2b4608b70144063a8a4a6e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.245698 24714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e8d8fbd2d1249d5b5de5f189b0d03e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.254166 26041 consensus_queue.cc:1035] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.255446 26766 tablet_bootstrap.cc:492] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:15.255625 26765 tablet_bootstrap.cc:492] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:15.257325 26766 tablet_bootstrap.cc:654] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.257339 26765 tablet_bootstrap.cc:654] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.293962 26787 tablet_bootstrap.cc:492] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:15.299549 26781 tablet_bootstrap.cc:492] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:15.300570 26787 tablet_bootstrap.cc:654] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.300655 26782 tablet_bootstrap.cc:492] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:15.302940 26781 tablet_bootstrap.cc:654] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.303893 26766 tablet_bootstrap.cc:492] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:15.304184 26766 ts_tablet_manager.cc:1367] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.004s
I20240502 01:54:15.305011 26766 raft_consensus.cc:348] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.305325 26766 raft_consensus.cc:374] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.305406 26766 raft_consensus.cc:729] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.305720 26766 consensus_queue.cc:260] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.310365 26766 ts_tablet_manager.cc:1398] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:54:15.312842 26787 tablet_bootstrap.cc:492] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:15.313951 26782 tablet_bootstrap.cc:654] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.314057 26787 ts_tablet_manager.cc:1367] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20240502 01:54:15.315341 26766 tablet_bootstrap.cc:492] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:15.316483 26787 raft_consensus.cc:348] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.317510 26787 raft_consensus.cc:374] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.317680 26787 raft_consensus.cc:729] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.318740 26765 tablet_bootstrap.cc:492] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:15.318984 26765 ts_tablet_manager.cc:1367] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.063s user 0.010s sys 0.008s
I20240502 01:54:15.319041 26766 tablet_bootstrap.cc:654] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.319927 26765 raft_consensus.cc:348] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.320241 26765 raft_consensus.cc:374] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.320358 26765 raft_consensus.cc:729] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.320688 26765 consensus_queue.cc:260] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.323698 26765 ts_tablet_manager.cc:1398] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:54:15.324056 26765 tablet_bootstrap.cc:492] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:15.325700 26765 tablet_bootstrap.cc:654] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.318002 26787 consensus_queue.cc:260] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.336639 26787 ts_tablet_manager.cc:1398] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240502 01:54:15.338583 26782 tablet_bootstrap.cc:492] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:15.338879 26782 ts_tablet_manager.cc:1367] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.004s
I20240502 01:54:15.340278 26782 raft_consensus.cc:348] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.341105 26781 tablet_bootstrap.cc:492] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:15.341230 26782 raft_consensus.cc:374] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.341400 26781 ts_tablet_manager.cc:1367] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.000s
I20240502 01:54:15.341745 26782 raft_consensus.cc:729] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.342350 26781 raft_consensus.cc:348] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.342103 26782 consensus_queue.cc:260] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.343534 26781 raft_consensus.cc:374] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.343760 26781 raft_consensus.cc:729] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.344063 26781 consensus_queue.cc:260] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.344252 26782 ts_tablet_manager.cc:1398] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:54:15.345911 26781 ts_tablet_manager.cc:1398] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:15.347208 26781 tablet_bootstrap.cc:492] T a9fb0b24337a4175972db268aac5df34 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:15.349077 26781 tablet_bootstrap.cc:654] T a9fb0b24337a4175972db268aac5df34 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.352550 26765 tablet_bootstrap.cc:492] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:15.352803 26765 ts_tablet_manager.cc:1367] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20240502 01:54:15.353714 26765 raft_consensus.cc:348] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.354002 26765 raft_consensus.cc:374] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.354130 26765 raft_consensus.cc:729] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.354421 26765 consensus_queue.cc:260] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.355118 26765 ts_tablet_manager.cc:1398] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:15.364467 21712 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:15.369108 26766 tablet_bootstrap.cc:492] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:15.369340 26766 ts_tablet_manager.cc:1367] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.000s
I20240502 01:54:15.375669 25000 raft_consensus.cc:1261] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:15.377115 21988 raft_consensus.cc:1261] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:15.377472 26537 consensus_queue.cc:1035] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.378031 26537 consensus_queue.cc:1035] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.370726 26766 raft_consensus.cc:348] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.389799 26766 raft_consensus.cc:374] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.389889 26766 raft_consensus.cc:729] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.391491 26781 tablet_bootstrap.cc:492] T a9fb0b24337a4175972db268aac5df34 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:15.391693 26781 ts_tablet_manager.cc:1367] T a9fb0b24337a4175972db268aac5df34 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.008s
I20240502 01:54:15.393371 21712 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 904a7757410c41f4af5b56ea661f2371 (table rebalancer_dynamic_table [id=2cd4d985b71d4cf0a0a6fc503d57c436]). Expected version 1 got 0
I20240502 01:54:15.393985 26781 raft_consensus.cc:348] T a9fb0b24337a4175972db268aac5df34 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.394304 26781 raft_consensus.cc:374] T a9fb0b24337a4175972db268aac5df34 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.394376 26781 raft_consensus.cc:729] T a9fb0b24337a4175972db268aac5df34 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.394667 26781 consensus_queue.cc:260] T a9fb0b24337a4175972db268aac5df34 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.395309 26781 ts_tablet_manager.cc:1398] T a9fb0b24337a4175972db268aac5df34 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:15.409098 26781 tablet_bootstrap.cc:492] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:15.410167 21715 catalog_manager.cc:5576] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411) does not have the latest schema for tablet 904a7757410c41f4af5b56ea661f2371 (table rebalancer_dynamic_table [id=2cd4d985b71d4cf0a0a6fc503d57c436]). Expected version 1 got 0
I20240502 01:54:15.423638 26781 tablet_bootstrap.cc:654] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.429540 26805 tablet.cc:1722] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:15.432811 26805 tablet.cc:1620] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412: MemRowSet was empty: no flush needed.
I20240502 01:54:15.434278 21710 catalog_manager.cc:5576] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411) does not have the latest schema for tablet 904a7757410c41f4af5b56ea661f2371 (table rebalancer_dynamic_table [id=2cd4d985b71d4cf0a0a6fc503d57c436]). Expected version 1 got 0
I20240502 01:54:15.442032 26781 tablet_bootstrap.cc:492] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:15.442312 26781 ts_tablet_manager.cc:1367] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.001s
I20240502 01:54:15.443185 26781 raft_consensus.cc:348] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.443486 26781 raft_consensus.cc:374] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.443634 26781 raft_consensus.cc:729] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.443956 26781 consensus_queue.cc:260] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.443872 26766 consensus_queue.cc:260] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.446132 26781 ts_tablet_manager.cc:1398] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:54:15.452142 21710 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=2cd4d985b71d4cf0a0a6fc503d57c436] alter complete (version 1)
I20240502 01:54:15.463712 26766 ts_tablet_manager.cc:1398] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.094s user 0.003s sys 0.000s
I20240502 01:54:15.470484 26803 tablet.cc:1722] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:15.469946 26802 tablet.cc:1722] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:15.474049 26803 tablet.cc:1620] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:15.474311 26802 tablet.cc:1620] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:15.484046 21712 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 904a7757410c41f4af5b56ea661f2371 (table rebalancer_dynamic_table [id=2cd4d985b71d4cf0a0a6fc503d57c436]). Expected version 1 got 0
I20240502 01:54:15.501871 26691 raft_consensus.cc:482] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:15.502058 26691 raft_consensus.cc:504] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.502974 26691 leader_election.cc:288] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:15.504957 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8d8fbd2d1249d5b5de5f189b0d03e9" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:15.505226 21857 raft_consensus.cc:2454] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:15.505714 24734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8d8fbd2d1249d5b5de5f189b0d03e9" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:15.505977 24734 raft_consensus.cc:2454] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:15.506551 26786 raft_consensus.cc:482] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:15.506673 24934 leader_election.cc:302] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:15.506767 26786 raft_consensus.cc:504] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.507613 26786 leader_election.cc:288] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:15.509833 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fb0b24337a4175972db268aac5df34" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:15.510083 24602 raft_consensus.cc:2454] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:15.511111 26691 raft_consensus.cc:2781] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:15.511256 26691 raft_consensus.cc:482] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:15.511354 26691 raft_consensus.cc:3037] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.513563 21923 leader_election.cc:302] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 0c64d40b5d7644bd8b29feaa80eab72f; no voters:
I20240502 01:54:15.513370 24734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fb0b24337a4175972db268aac5df34" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:15.514081 26786 raft_consensus.cc:2781] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:15.514212 26786 raft_consensus.cc:482] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:15.514062 24734 raft_consensus.cc:2454] T a9fb0b24337a4175972db268aac5df34 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:15.514173 26691 raft_consensus.cc:504] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.514302 26786 raft_consensus.cc:3037] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.515444 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8d8fbd2d1249d5b5de5f189b0d03e9" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:15.515677 21857 raft_consensus.cc:3037] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.517304 26786 raft_consensus.cc:504] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.518112 26786 leader_election.cc:288] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:15.518499 21857 raft_consensus.cc:2454] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:15.519331 24933 leader_election.cc:302] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:15.519718 26692 raft_consensus.cc:2781] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:15.519716 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fb0b24337a4175972db268aac5df34" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:15.519959 24602 raft_consensus.cc:3037] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.523530 24734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fb0b24337a4175972db268aac5df34" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:15.523806 24734 raft_consensus.cc:3037] T a9fb0b24337a4175972db268aac5df34 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.527648 24602 raft_consensus.cc:2454] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:15.528028 26429 raft_consensus.cc:482] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:15.528276 24733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8d8fbd2d1249d5b5de5f189b0d03e9" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:15.528223 26429 raft_consensus.cc:504] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.528409 26691 leader_election.cc:288] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:15.528466 24733 raft_consensus.cc:3037] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.529011 26429 leader_election.cc:288] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:15.530792 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafa4ef17fac41b68e3bf4aaa8a4caa8" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:15.531020 25000 raft_consensus.cc:2454] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:15.531199 24733 raft_consensus.cc:2454] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:15.532153 21925 leader_election.cc:302] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:15.533439 26429 raft_consensus.cc:2781] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:15.533547 26429 raft_consensus.cc:482] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:15.533543 24733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafa4ef17fac41b68e3bf4aaa8a4caa8" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:15.533660 26429 raft_consensus.cc:3037] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.533769 24733 raft_consensus.cc:2454] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:15.540760 26692 raft_consensus.cc:686] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:54:15.541311 26692 consensus_queue.cc:237] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.541550 21923 leader_election.cc:302] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 0c64d40b5d7644bd8b29feaa80eab72f; no voters:
I20240502 01:54:15.542135 26786 raft_consensus.cc:2781] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:15.542765 24734 raft_consensus.cc:2454] T a9fb0b24337a4175972db268aac5df34 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:15.543287 26429 raft_consensus.cc:504] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.545105 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafa4ef17fac41b68e3bf4aaa8a4caa8" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:15.545334 25001 raft_consensus.cc:3037] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.545652 24733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafa4ef17fac41b68e3bf4aaa8a4caa8" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:15.545872 24733 raft_consensus.cc:3037] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.548214 25001 raft_consensus.cc:2454] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:15.548367 21712 catalog_manager.cc:5484] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:15.548676 24733 raft_consensus.cc:2454] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:15.550295 26786 raft_consensus.cc:686] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:54:15.550511 21925 leader_election.cc:302] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:15.550697 26786 consensus_queue.cc:237] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.550998 26811 raft_consensus.cc:2781] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:15.551167 26811 raft_consensus.cc:686] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:54:15.551493 26811 consensus_queue.cc:237] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.553211 26429 leader_election.cc:288] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:15.555913 21712 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:15.565032 21712 catalog_manager.cc:5484] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:15.565824 26802 tablet.cc:1722] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:15.566354 26802 tablet.cc:1620] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:15.566630 26805 tablet.cc:1722] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:15.567126 26805 tablet.cc:1620] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412: MemRowSet was empty: no flush needed.
I20240502 01:54:15.570333 26803 tablet.cc:1722] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:15.570796 26803 tablet.cc:1620] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:15.575067 21712 catalog_manager.cc:5484] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:15.587689 21713 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=2cd4d985b71d4cf0a0a6fc503d57c436] alter complete (version 2)
I20240502 01:54:15.598992 22117 consensus_queue.cc:237] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:15.601224 21857 raft_consensus.cc:1261] T 166e717fb9624ae58adb6699790daf7a P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:15.601260 24733 raft_consensus.cc:1261] T 166e717fb9624ae58adb6699790daf7a P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:15.601958 26647 consensus_queue.cc:1035] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:15.602352 26648 consensus_queue.cc:1035] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:15.605839 26647 raft_consensus.cc:2932] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:15.607749 24733 raft_consensus.cc:2932] T 166e717fb9624ae58adb6699790daf7a P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:15.608397 21857 raft_consensus.cc:2932] T 166e717fb9624ae58adb6699790daf7a P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:15.614156 21715 catalog_manager.cc:5484] T 166e717fb9624ae58adb6699790daf7a P 28157bc3e678428fb5f93c077d528513 reported cstate change: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:15.621440 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet 166e717fb9624ae58adb6699790daf7a with delete_type TABLET_DATA_TOMBSTONED (TS c2a28baa7ee2469294d94b7fe5455cff not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:15.629565 26831 tablet_replica.cc:330] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:15.629982 26831 raft_consensus.cc:2229] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:15.630694 26831 raft_consensus.cc:2258] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:15.631472 26831 ts_tablet_manager.cc:1875] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:15.637621 26831 ts_tablet_manager.cc:1888] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:54:15.637868 26831 log.cc:1196] T 166e717fb9624ae58adb6699790daf7a P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/166e717fb9624ae58adb6699790daf7a
I20240502 01:54:15.638715 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 166e717fb9624ae58adb6699790daf7a (table rebalance_test_table_9 [id=714c5d94b5b64cb29dc398713b1727fe]) successfully deleted
I20240502 01:54:15.691670 21712 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:15.703776 26802 tablet.cc:1722] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:15.704608 26805 tablet.cc:1722] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:15.709721 26803 tablet.cc:1722] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:15.717850 21712 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=2cd4d985b71d4cf0a0a6fc503d57c436] alter complete (version 3)
I20240502 01:54:15.829746 21707 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:15.830080 21707 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:15.843758 21707 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 904a7757410c41f4af5b56ea661f2371
I20240502 01:54:15.844956 24846 tablet_service.cc:1514] Processing DeleteTablet for tablet 904a7757410c41f4af5b56ea661f2371 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:15 UTC) from {username='slave'} at 127.0.0.1:51592
I20240502 01:54:15.845981 21967 tablet_service.cc:1514] Processing DeleteTablet for tablet 904a7757410c41f4af5b56ea661f2371 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:15 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:15.847374 24979 tablet_service.cc:1514] Processing DeleteTablet for tablet 904a7757410c41f4af5b56ea661f2371 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:15 UTC) from {username='slave'} at 127.0.0.1:39964
I20240502 01:54:15.861891 26872 tablet_replica.cc:330] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:15.863847 26872 raft_consensus.cc:2229] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:15.864213 26872 raft_consensus.cc:2258] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:15.865289 26872 ts_tablet_manager.cc:1875] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:15.871881 26872 ts_tablet_manager.cc:1888] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:15.872104 26872 log.cc:1196] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/904a7757410c41f4af5b56ea661f2371
I20240502 01:54:15.872592 26872 ts_tablet_manager.cc:1909] T 904a7757410c41f4af5b56ea661f2371 P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:15.873906 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 904a7757410c41f4af5b56ea661f2371 (table rebalancer_dynamic_table_new_name [id=2cd4d985b71d4cf0a0a6fc503d57c436]) successfully deleted
I20240502 01:54:15.896342 26873 tablet_replica.cc:330] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:54:15.897099 26871 tablet_replica.cc:330] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08: stopping tablet replica
I20240502 01:54:15.897490 26871 raft_consensus.cc:2229] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:15.897882 26873 raft_consensus.cc:2229] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:15.898504 26871 raft_consensus.cc:2258] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:15.899498 26873 raft_consensus.cc:2258] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:15.900914 26873 ts_tablet_manager.cc:1875] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:15.902050 26871 ts_tablet_manager.cc:1875] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:15.921615 26871 ts_tablet_manager.cc:1888] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:15.922413 26871 log.cc:1196] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/wals/904a7757410c41f4af5b56ea661f2371
I20240502 01:54:15.922988 26871 ts_tablet_manager.cc:1909] T 904a7757410c41f4af5b56ea661f2371 P d7449bb1df214bf4a5f93e4631312f08: Deleting consensus metadata
I20240502 01:54:15.925379 21700 catalog_manager.cc:4830] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): tablet 904a7757410c41f4af5b56ea661f2371 (table rebalancer_dynamic_table_new_name [id=2cd4d985b71d4cf0a0a6fc503d57c436]) successfully deleted
I20240502 01:54:15.938570 26873 ts_tablet_manager.cc:1888] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:15.938867 26873 log.cc:1196] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/904a7757410c41f4af5b56ea661f2371
I20240502 01:54:15.939527 26873 ts_tablet_manager.cc:1909] T 904a7757410c41f4af5b56ea661f2371 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting consensus metadata
I20240502 01:54:15.941059 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet 904a7757410c41f4af5b56ea661f2371 (table rebalancer_dynamic_table_new_name [id=2cd4d985b71d4cf0a0a6fc503d57c436]) successfully deleted
I20240502 01:54:15.948421 21707 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:15.964138 26786 consensus_queue.cc:1035] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.973601 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 234810c749c5481f99a2e6fe0fddbe27 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b2072f0889b749fab0f6e10b81e845d2]), partition=
I20240502 01:54:15.973613 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 234810c749c5481f99a2e6fe0fddbe27 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b2072f0889b749fab0f6e10b81e845d2]), partition=
I20240502 01:54:15.973654 24979 tablet_service.cc:1467] Processing CreateTablet for tablet 234810c749c5481f99a2e6fe0fddbe27 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b2072f0889b749fab0f6e10b81e845d2]), partition=
I20240502 01:54:15.974073 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 234810c749c5481f99a2e6fe0fddbe27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.974073 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 234810c749c5481f99a2e6fe0fddbe27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.974097 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 234810c749c5481f99a2e6fe0fddbe27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.985957 26786 consensus_queue.cc:1035] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.997740 21967 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:15.999956 26885 tablet_bootstrap.cc:492] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:16.004446 26887 tablet_bootstrap.cc:492] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:16.007637 26884 tablet_bootstrap.cc:492] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:16.013751 26884 tablet_bootstrap.cc:654] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.013603 26811 consensus_queue.cc:1035] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.008872 26885 tablet_bootstrap.cc:654] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.029976 26884 tablet_bootstrap.cc:492] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:16.030297 26884 ts_tablet_manager.cc:1367] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.006s
I20240502 01:54:16.034029 26885 tablet_bootstrap.cc:492] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:16.034324 26885 ts_tablet_manager.cc:1367] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.000s
I20240502 01:54:16.035243 26885 raft_consensus.cc:348] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.035591 26885 raft_consensus.cc:374] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.035739 26885 raft_consensus.cc:729] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.036082 26885 consensus_queue.cc:260] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.038311 26885 ts_tablet_manager.cc:1398] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:16.031689 26884 raft_consensus.cc:348] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.042150 26884 raft_consensus.cc:374] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.042330 26884 raft_consensus.cc:729] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.042500 26807 consensus_queue.cc:1035] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.043231 26884 consensus_queue.cc:260] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.044661 26884 ts_tablet_manager.cc:1398] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240502 01:54:16.057374 26812 consensus_queue.cc:1035] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.057631 24583 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:16.061537 26887 tablet_bootstrap.cc:654] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.115170 24713 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:16.133145 26905 consensus_queue.cc:1035] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 01:54:16.135684 26887 tablet_bootstrap.cc:492] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:16.136412 26887 ts_tablet_manager.cc:1367] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.132s user 0.005s sys 0.000s
I20240502 01:54:16.137570 26887 raft_consensus.cc:348] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.137916 26887 raft_consensus.cc:374] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.138105 26887 raft_consensus.cc:729] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.140167 21836 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:16.157958 21707 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:16.163152 22231 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:16.171438 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 83480ad46a764f29a4fdfd782151ee43 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=3f753f8c45ae4fc89dfc2c1572a89224]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.172013 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83480ad46a764f29a4fdfd782151ee43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.179539 24846 tablet_service.cc:1467] Processing CreateTablet for tablet 1803e35563e84c84b90cbf608ecf6a93 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=3f753f8c45ae4fc89dfc2c1572a89224]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.180343 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1803e35563e84c84b90cbf608ecf6a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.182127 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 1803e35563e84c84b90cbf608ecf6a93 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=3f753f8c45ae4fc89dfc2c1572a89224]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.182550 24848 tablet_service.cc:1467] Processing CreateTablet for tablet 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=3f753f8c45ae4fc89dfc2c1572a89224]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.182736 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1803e35563e84c84b90cbf608ecf6a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.183058 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28dbb8d9ef5f4efdb5ba6e8ba9179aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.183753 24713 tablet_service.cc:1467] Processing CreateTablet for tablet 1803e35563e84c84b90cbf608ecf6a93 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=3f753f8c45ae4fc89dfc2c1572a89224]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.184325 24713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1803e35563e84c84b90cbf608ecf6a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.188158 21967 tablet_service.cc:1467] Processing CreateTablet for tablet 83480ad46a764f29a4fdfd782151ee43 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=3f753f8c45ae4fc89dfc2c1572a89224]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.188711 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83480ad46a764f29a4fdfd782151ee43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.195402 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=3f753f8c45ae4fc89dfc2c1572a89224]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.196058 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28dbb8d9ef5f4efdb5ba6e8ba9179aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.198385 26884 tablet_bootstrap.cc:492] T 83480ad46a764f29a4fdfd782151ee43 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:16.201140 26884 tablet_bootstrap.cc:654] T 83480ad46a764f29a4fdfd782151ee43 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.211944 26914 tablet_bootstrap.cc:492] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:16.212293 26884 tablet_bootstrap.cc:492] T 83480ad46a764f29a4fdfd782151ee43 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:16.212571 26884 ts_tablet_manager.cc:1367] T 83480ad46a764f29a4fdfd782151ee43 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20240502 01:54:16.214995 26910 tablet_bootstrap.cc:492] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:16.214982 26884 raft_consensus.cc:348] T 83480ad46a764f29a4fdfd782151ee43 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.215870 26911 tablet_bootstrap.cc:492] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:16.215495 26884 raft_consensus.cc:374] T 83480ad46a764f29a4fdfd782151ee43 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.217602 26884 raft_consensus.cc:729] T 83480ad46a764f29a4fdfd782151ee43 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.218142 26910 tablet_bootstrap.cc:654] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.218084 26884 consensus_queue.cc:260] T 83480ad46a764f29a4fdfd782151ee43 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.219054 26913 tablet_bootstrap.cc:492] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:16.223474 26910 tablet_bootstrap.cc:492] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:16.223842 26910 ts_tablet_manager.cc:1367] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240502 01:54:16.224014 26914 tablet_bootstrap.cc:654] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.225467 26910 raft_consensus.cc:348] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:16.225916 26910 raft_consensus.cc:374] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.226169 26910 raft_consensus.cc:729] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.226806 26910 consensus_queue.cc:260] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:16.227865 26910 ts_tablet_manager.cc:1398] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:16.230089 26911 tablet_bootstrap.cc:654] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.234072 26913 tablet_bootstrap.cc:654] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.239498 26884 ts_tablet_manager.cc:1398] T 83480ad46a764f29a4fdfd782151ee43 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20240502 01:54:16.154893 24979 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 01:54:16.261716 24934 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.198:42375, user_credentials={real_user=slave}} blocked reactor thread for 138761us
I20240502 01:54:16.263291 24979 tablet_service.cc:1467] Processing CreateTablet for tablet 83480ad46a764f29a4fdfd782151ee43 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=3f753f8c45ae4fc89dfc2c1572a89224]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.263674 24981 tablet_service.cc:1467] Processing CreateTablet for tablet 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=3f753f8c45ae4fc89dfc2c1572a89224]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.263953 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83480ad46a764f29a4fdfd782151ee43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.264328 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28dbb8d9ef5f4efdb5ba6e8ba9179aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.267094 26906 raft_consensus.cc:482] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:16.267321 26906 raft_consensus.cc:504] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.271632 26906 leader_election.cc:288] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:16.271828 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234810c749c5481f99a2e6fe0fddbe27" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:16.272982 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234810c749c5481f99a2e6fe0fddbe27" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:16.273381 24602 raft_consensus.cc:2454] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:16.273699 26887 consensus_queue.cc:260] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.274446 21856 raft_consensus.cc:2454] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:16.275460 24933 leader_election.cc:302] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:16.275854 26906 raft_consensus.cc:2781] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:16.276154 26906 raft_consensus.cc:482] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:16.276291 26906 raft_consensus.cc:3037] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.278735 26887 ts_tablet_manager.cc:1398] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.142s user 0.003s sys 0.000s
I20240502 01:54:16.283308 26914 tablet_bootstrap.cc:492] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:16.283684 26914 ts_tablet_manager.cc:1367] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.072s user 0.005s sys 0.007s
I20240502 01:54:16.285104 26914 raft_consensus.cc:348] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.285573 26914 raft_consensus.cc:374] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.286196 26914 raft_consensus.cc:729] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.286940 26914 consensus_queue.cc:260] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.291606 26914 ts_tablet_manager.cc:1398] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20240502 01:54:16.323540 26911 tablet_bootstrap.cc:492] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:16.323792 26911 ts_tablet_manager.cc:1367] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.108s user 0.004s sys 0.018s
I20240502 01:54:16.325111 26911 raft_consensus.cc:348] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:16.325392 26911 raft_consensus.cc:374] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.325592 26911 raft_consensus.cc:729] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.325938 26911 consensus_queue.cc:260] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:16.329058 26911 ts_tablet_manager.cc:1398] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:54:16.331887 26911 tablet_bootstrap.cc:492] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
W20240502 01:54:16.338958 21792 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.195:38879, user_credentials={real_user=slave}} blocked reactor thread for 71666.8us
I20240502 01:54:16.341953 26911 tablet_bootstrap.cc:654] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.362478 24846 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:16.381736 26915 tablet_bootstrap.cc:492] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:16.388700 26915 tablet_bootstrap.cc:654] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.399430 26913 tablet_bootstrap.cc:492] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:16.399940 26913 ts_tablet_manager.cc:1367] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.181s user 0.000s sys 0.007s
I20240502 01:54:16.429183 26906 raft_consensus.cc:504] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.430656 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234810c749c5481f99a2e6fe0fddbe27" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:16.430970 24603 raft_consensus.cc:3037] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.431092 26812 raft_consensus.cc:482] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:16.431304 26812 raft_consensus.cc:504] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.432161 26812 leader_election.cc:288] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:16.433188 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83480ad46a764f29a4fdfd782151ee43" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:16.433454 24602 raft_consensus.cc:2454] T 83480ad46a764f29a4fdfd782151ee43 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:16.434100 21923 leader_election.cc:302] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 0c64d40b5d7644bd8b29feaa80eab72f; no voters:
I20240502 01:54:16.434664 26812 raft_consensus.cc:2781] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:16.434881 26812 raft_consensus.cc:482] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:16.435034 26812 raft_consensus.cc:3037] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.438196 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234810c749c5481f99a2e6fe0fddbe27" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:16.438540 26812 raft_consensus.cc:504] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.438807 21854 raft_consensus.cc:3037] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.440022 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83480ad46a764f29a4fdfd782151ee43" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:16.440328 24602 raft_consensus.cc:3037] T 83480ad46a764f29a4fdfd782151ee43 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.445185 24602 raft_consensus.cc:2454] T 83480ad46a764f29a4fdfd782151ee43 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:16.445775 21923 leader_election.cc:302] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 0c64d40b5d7644bd8b29feaa80eab72f; no voters:
I20240502 01:54:16.446193 26811 raft_consensus.cc:2781] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:16.446350 26811 raft_consensus.cc:686] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:54:16.446750 26811 consensus_queue.cc:237] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.449925 21854 raft_consensus.cc:2454] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:16.451315 24603 raft_consensus.cc:2454] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:16.451342 26812 leader_election.cc:288] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:16.453008 24934 leader_election.cc:302] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:16.454425 26906 leader_election.cc:288] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:16.455408 21712 catalog_manager.cc:5484] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
W20240502 01:54:16.461612 21925 leader_election.cc:341] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Not found: Tablet not found: 83480ad46a764f29a4fdfd782151ee43
W20240502 01:54:16.462157 21925 leader_election.cc:341] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Not found: Tablet not found: 83480ad46a764f29a4fdfd782151ee43
I20240502 01:54:16.474900 26885 tablet_bootstrap.cc:492] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:16.476717 26906 raft_consensus.cc:2781] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:16.477092 26906 raft_consensus.cc:686] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:54:16.484421 26885 tablet_bootstrap.cc:654] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.486423 26906 consensus_queue.cc:237] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.495884 26885 tablet_bootstrap.cc:492] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:16.496187 26885 ts_tablet_manager.cc:1367] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.011s
I20240502 01:54:16.515532 26913 raft_consensus.cc:348] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.515987 26913 raft_consensus.cc:374] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.516172 26913 raft_consensus.cc:729] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.516551 26913 consensus_queue.cc:260] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.517585 26913 ts_tablet_manager.cc:1398] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.117s user 0.003s sys 0.000s
I20240502 01:54:16.525517 26911 tablet_bootstrap.cc:492] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:16.525386 26885 raft_consensus.cc:348] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.526057 26885 raft_consensus.cc:374] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.526269 26885 raft_consensus.cc:729] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.526743 26885 consensus_queue.cc:260] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.527706 26885 ts_tablet_manager.cc:1398] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.031s user 0.000s sys 0.003s
I20240502 01:54:16.527778 26911 ts_tablet_manager.cc:1367] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.196s user 0.007s sys 0.023s
I20240502 01:54:16.528343 26885 tablet_bootstrap.cc:492] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
W20240502 01:54:16.536170 22056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.200:37097, user_credentials={real_user=slave}} blocked reactor thread for 101142us
I20240502 01:54:16.532209 26911 raft_consensus.cc:348] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.539294 26911 raft_consensus.cc:374] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.539451 26911 raft_consensus.cc:729] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.539835 26911 consensus_queue.cc:260] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.540666 26911 ts_tablet_manager.cc:1398] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 01:54:16.559569 22100 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:16.571429 26915 tablet_bootstrap.cc:492] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:16.571689 26915 ts_tablet_manager.cc:1367] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.196s user 0.010s sys 0.027s
I20240502 01:54:16.572911 26915 raft_consensus.cc:348] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:16.573174 26915 raft_consensus.cc:374] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.573372 26915 raft_consensus.cc:729] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.573721 26915 consensus_queue.cc:260] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:16.578667 26537 raft_consensus.cc:482] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:16.578896 26537 raft_consensus.cc:504] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:16.579888 26537 leader_election.cc:288] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:16.581918 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1803e35563e84c84b90cbf608ecf6a93" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:16.582253 24732 raft_consensus.cc:2454] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:16.592871 21707 catalog_manager.cc:5484] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:16.594242 24802 leader_election.cc:302] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:16.598572 26924 raft_consensus.cc:2781] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:16.598778 26924 raft_consensus.cc:482] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:16.598955 26924 raft_consensus.cc:3037] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.604809 26915 ts_tablet_manager.cc:1398] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.033s user 0.000s sys 0.003s
I20240502 01:54:16.608786 26924 raft_consensus.cc:504] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:16.610229 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1803e35563e84c84b90cbf608ecf6a93" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:16.610517 24735 raft_consensus.cc:3037] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.618743 26885 tablet_bootstrap.cc:654] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.619066 24735 raft_consensus.cc:2454] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:16.619622 24802 leader_election.cc:302] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:16.620352 26924 leader_election.cc:288] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:16.620596 26924 raft_consensus.cc:2781] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:16.620807 26924 raft_consensus.cc:686] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:16.621173 26924 consensus_queue.cc:237] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:16.632596 26934 raft_consensus.cc:482] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:16.633028 26934 raft_consensus.cc:504] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:16.634546 26934 leader_election.cc:288] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:16.638384 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1803e35563e84c84b90cbf608ecf6a93" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:16.638782 24732 raft_consensus.cc:2379] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c2a28baa7ee2469294d94b7fe5455cff in current term 1: Already voted for candidate d7449bb1df214bf4a5f93e4631312f08 in this term.
I20240502 01:54:16.638788 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1803e35563e84c84b90cbf608ecf6a93" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:16.642634 21707 catalog_manager.cc:5484] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:16.648659 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1803e35563e84c84b90cbf608ecf6a93" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:16.648986 22249 raft_consensus.cc:2454] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:16.640170 22185 leader_election.cc:302] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:16.649804 26936 raft_consensus.cc:3037] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.652827 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1803e35563e84c84b90cbf608ecf6a93" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:16.653195 26647 raft_consensus.cc:482] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:16.653391 26647 raft_consensus.cc:504] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.654469 26647 leader_election.cc:288] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:16.655398 26936 raft_consensus.cc:2726] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:16.660331 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28dbb8d9ef5f4efdb5ba6e8ba9179aa8" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:16.660619 24867 raft_consensus.cc:2454] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:54:16.660739 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28dbb8d9ef5f4efdb5ba6e8ba9179aa8" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:16.660992 24999 raft_consensus.cc:2454] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:54:16.661437 26885 tablet_bootstrap.cc:492] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:16.661602 26885 ts_tablet_manager.cc:1367] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.133s user 0.006s sys 0.000s
I20240502 01:54:16.661592 22056 leader_election.cc:302] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:54:16.661885 22250 raft_consensus.cc:2454] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:16.662071 26928 raft_consensus.cc:2781] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:16.662184 26928 raft_consensus.cc:482] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:16.662307 26928 raft_consensus.cc:3037] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.662642 26885 raft_consensus.cc:348] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.662878 26885 raft_consensus.cc:374] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.663472 26885 raft_consensus.cc:729] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.663722 26885 consensus_queue.cc:260] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.664322 26885 ts_tablet_manager.cc:1398] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:16.665613 26928 raft_consensus.cc:504] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.667479 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28dbb8d9ef5f4efdb5ba6e8ba9179aa8" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:16.667680 24867 raft_consensus.cc:3037] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.670759 24867 raft_consensus.cc:2454] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:54:16.671288 22054 leader_election.cc:302] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:16.671695 26647 raft_consensus.cc:2781] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:16.671856 26647 raft_consensus.cc:686] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:54:16.672107 26647 consensus_queue.cc:237] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.674319 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28dbb8d9ef5f4efdb5ba6e8ba9179aa8" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:16.674626 24999 raft_consensus.cc:3037] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:54:16.674890 22184 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.193:33785, user_credentials={real_user=slave}} blocked reactor thread for 285570us
I20240502 01:54:16.678344 26928 leader_election.cc:288] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:16.679106 24999 raft_consensus.cc:2454] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:54:16.693457 21707 catalog_manager.cc:5484] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:16.728040 21707 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:16.744042 24603 raft_consensus.cc:1261] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:16.744380 21854 raft_consensus.cc:1261] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:16.744748 26906 consensus_queue.cc:1035] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.745040 26905 consensus_queue.cc:1035] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.751219 21712 catalog_manager.cc:5576] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785) does not have the latest schema for tablet 234810c749c5481f99a2e6fe0fddbe27 (table rebalancer_dynamic_table [id=b2072f0889b749fab0f6e10b81e845d2]). Expected version 1 got 0
I20240502 01:54:16.754361 26921 mvcc.cc:205] Tried to move back new op lower bound from 7023062453216206848 to 7023062452186476544. Current Snapshot: MvccSnapshot[applied={T|T < 7023062453216206848}]
I20240502 01:54:16.756593 26905 raft_consensus.cc:482] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:16.756809 26905 raft_consensus.cc:504] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.754778 26946 tablet.cc:1722] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:16.757793 26905 leader_election.cc:288] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:16.758162 26946 tablet.cc:1620] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77: MemRowSet was empty: no flush needed.
I20240502 01:54:16.759353 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83480ad46a764f29a4fdfd782151ee43" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:54:16.759572 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83480ad46a764f29a4fdfd782151ee43" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:16.759960 24602 raft_consensus.cc:2379] T 83480ad46a764f29a4fdfd782151ee43 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 41c70f373c9d49a49891b03a12c6cb2b in current term 1: Already voted for candidate 06543b05a16f42099bc86db2a0284412 in this term.
I20240502 01:54:16.760473 21712 catalog_manager.cc:5576] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785) does not have the latest schema for tablet 234810c749c5481f99a2e6fe0fddbe27 (table rebalancer_dynamic_table [id=b2072f0889b749fab0f6e10b81e845d2]). Expected version 1 got 0
I20240502 01:54:16.762306 24933 leader_election.cc:302] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 06543b05a16f42099bc86db2a0284412, 0c64d40b5d7644bd8b29feaa80eab72f
I20240502 01:54:16.763804 26905 raft_consensus.cc:3037] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.767783 26905 raft_consensus.cc:2726] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:16.769268 26945 tablet.cc:1722] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:16.774139 21712 catalog_manager.cc:5576] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677) does not have the latest schema for tablet 234810c749c5481f99a2e6fe0fddbe27 (table rebalancer_dynamic_table [id=b2072f0889b749fab0f6e10b81e845d2]). Expected version 1 got 0
I20240502 01:54:16.770438 26945 tablet.cc:1620] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:16.777060 21707 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=b2072f0889b749fab0f6e10b81e845d2] alter complete (version 1)
I20240502 01:54:16.779326 26950 tablet.cc:1722] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:16.780855 26950 tablet.cc:1620] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:16.782003 21707 catalog_manager.cc:5576] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677) does not have the latest schema for tablet 234810c749c5481f99a2e6fe0fddbe27 (table rebalancer_dynamic_table [id=b2072f0889b749fab0f6e10b81e845d2]). Expected version 1 got 0
I20240502 01:54:16.814006 22117 consensus_queue.cc:237] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:16.816758 22250 raft_consensus.cc:1261] T ad3cd6fff977431b9cd5df6d083bef80 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:16.817551 24999 raft_consensus.cc:1261] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:16.818001 26928 consensus_queue.cc:1035] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:16.818387 26647 consensus_queue.cc:1035] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:16.826726 26928 raft_consensus.cc:2932] T ad3cd6fff977431b9cd5df6d083bef80 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:16.831149 22250 raft_consensus.cc:2932] T ad3cd6fff977431b9cd5df6d083bef80 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:16.837249 21707 catalog_manager.cc:5484] T ad3cd6fff977431b9cd5df6d083bef80 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:16.841638 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet ad3cd6fff977431b9cd5df6d083bef80 with delete_type TABLET_DATA_TOMBSTONED (TS 759257ea7bef415ca698e60d82199d77 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:16.852064 24999 raft_consensus.cc:2932] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:16.862083 26965 tablet_replica.cc:330] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:16.862417 26965 raft_consensus.cc:2229] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:16.862852 26965 raft_consensus.cc:2258] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:16.863881 26965 ts_tablet_manager.cc:1875] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:16.879734 26965 ts_tablet_manager.cc:1888] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:54:16.879992 26965 log.cc:1196] T ad3cd6fff977431b9cd5df6d083bef80 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/ad3cd6fff977431b9cd5df6d083bef80
I20240502 01:54:16.880939 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet ad3cd6fff977431b9cd5df6d083bef80 (table rebalance_test_table_17 [id=7a17c6db79004eb696e7dd281fdc4f3b]) successfully deleted
I20240502 01:54:16.886221 21707 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:16.894201 26812 consensus_queue.cc:1035] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.899957 26945 tablet.cc:1722] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:16.900441 26945 tablet.cc:1620] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:16.906481 26950 tablet.cc:1722] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:16.906997 26950 tablet.cc:1620] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:16.916798 21707 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=b2072f0889b749fab0f6e10b81e845d2] alter complete (version 2)
I20240502 01:54:16.927318 26946 tablet.cc:1722] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:16.927839 26946 tablet.cc:1620] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77: MemRowSet was empty: no flush needed.
I20240502 01:54:16.934947 26811 consensus_queue.cc:1035] T 83480ad46a764f29a4fdfd782151ee43 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.999522 26924 consensus_queue.cc:1035] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:17.014209 26537 consensus_queue.cc:1035] T 1803e35563e84c84b90cbf608ecf6a93 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:17.021510 21712 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:17.030196 26945 tablet.cc:1722] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:17.035636 26946 tablet.cc:1722] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:17.036305 26950 tablet.cc:1722] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:17.046813 21707 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=b2072f0889b749fab0f6e10b81e845d2] alter complete (version 3)
I20240502 01:54:17.150791 21709 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:17.151046 21709 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:17.157469 21709 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 234810c749c5481f99a2e6fe0fddbe27
I20240502 01:54:17.158596 24583 tablet_service.cc:1514] Processing DeleteTablet for tablet 234810c749c5481f99a2e6fe0fddbe27 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:17 UTC) from {username='slave'} at 127.0.0.1:53832
I20240502 01:54:17.165798 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 234810c749c5481f99a2e6fe0fddbe27 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:17 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:17.166132 26965 tablet_replica.cc:330] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:17.166092 24979 tablet_service.cc:1514] Processing DeleteTablet for tablet 234810c749c5481f99a2e6fe0fddbe27 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:17 UTC) from {username='slave'} at 127.0.0.1:39964
I20240502 01:54:17.166342 26965 raft_consensus.cc:2229] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:17.166559 26965 raft_consensus.cc:2258] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:17.167577 27016 tablet_replica.cc:330] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:54:17.167717 27017 tablet_replica.cc:330] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:54:17.167872 27016 raft_consensus.cc:2229] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:17.167925 26965 ts_tablet_manager.cc:1875] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:17.167937 27017 raft_consensus.cc:2229] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:17.168141 27016 raft_consensus.cc:2258] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:17.168421 27017 raft_consensus.cc:2258] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:17.169082 27016 ts_tablet_manager.cc:1875] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:17.169409 27017 ts_tablet_manager.cc:1875] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:17.174626 26965 ts_tablet_manager.cc:1888] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:17.175067 26965 log.cc:1196] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/234810c749c5481f99a2e6fe0fddbe27
I20240502 01:54:17.175582 26965 ts_tablet_manager.cc:1909] T 234810c749c5481f99a2e6fe0fddbe27 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:17.176309 27017 ts_tablet_manager.cc:1888] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:17.176441 27017 log.cc:1196] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/234810c749c5481f99a2e6fe0fddbe27
I20240502 01:54:17.177170 27017 ts_tablet_manager.cc:1909] T 234810c749c5481f99a2e6fe0fddbe27 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting consensus metadata
I20240502 01:54:17.178256 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet 234810c749c5481f99a2e6fe0fddbe27 (table rebalancer_dynamic_table_new_name [id=b2072f0889b749fab0f6e10b81e845d2]) successfully deleted
I20240502 01:54:17.178624 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 234810c749c5481f99a2e6fe0fddbe27 (table rebalancer_dynamic_table_new_name [id=b2072f0889b749fab0f6e10b81e845d2]) successfully deleted
I20240502 01:54:17.192173 27016 ts_tablet_manager.cc:1888] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:17.192384 27016 log.cc:1196] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/234810c749c5481f99a2e6fe0fddbe27
I20240502 01:54:17.194739 27016 ts_tablet_manager.cc:1909] T 234810c749c5481f99a2e6fe0fddbe27 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting consensus metadata
I20240502 01:54:17.196528 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet 234810c749c5481f99a2e6fe0fddbe27 (table rebalancer_dynamic_table_new_name [id=b2072f0889b749fab0f6e10b81e845d2]) successfully deleted
I20240502 01:54:17.212594 26928 consensus_queue.cc:1035] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:17.218122 21709 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:17.231515 21967 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:17.234011 24713 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:17.239336 24583 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:17.243039 24846 tablet_service.cc:1467] Processing CreateTablet for tablet 2b0e8db8c3414ce0a5ff12200fe6b544 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=3bb44c54e77145c39bccee47a8077319]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:17.243484 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b0e8db8c3414ce0a5ff12200fe6b544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.246856 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 2b0e8db8c3414ce0a5ff12200fe6b544 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=3bb44c54e77145c39bccee47a8077319]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:17.247383 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b0e8db8c3414ce0a5ff12200fe6b544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.258198 24713 tablet_service.cc:1467] Processing CreateTablet for tablet 2b0e8db8c3414ce0a5ff12200fe6b544 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=3bb44c54e77145c39bccee47a8077319]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:17.258656 24713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b0e8db8c3414ce0a5ff12200fe6b544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.258697 24979 tablet_service.cc:1467] Processing CreateTablet for tablet ee09342ee0f54863a158cb76d2112086 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=3bb44c54e77145c39bccee47a8077319]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:17.259246 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee09342ee0f54863a158cb76d2112086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.264911 24848 tablet_service.cc:1467] Processing CreateTablet for tablet ee09342ee0f54863a158cb76d2112086 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=3bb44c54e77145c39bccee47a8077319]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:17.265347 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee09342ee0f54863a158cb76d2112086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.268013 21709 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:17.269304 22100 tablet_service.cc:1467] Processing CreateTablet for tablet ee09342ee0f54863a158cb76d2112086 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=3bb44c54e77145c39bccee47a8077319]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:17.269755 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee09342ee0f54863a158cb76d2112086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.275676 22098 tablet_service.cc:1467] Processing CreateTablet for tablet a2c3475bcad94e3dad4be87376e500bc (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=3bb44c54e77145c39bccee47a8077319]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:17.276110 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c3475bcad94e3dad4be87376e500bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.278692 24847 tablet_service.cc:1467] Processing CreateTablet for tablet a2c3475bcad94e3dad4be87376e500bc (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=3bb44c54e77145c39bccee47a8077319]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:17.278816 24715 tablet_service.cc:1467] Processing CreateTablet for tablet a2c3475bcad94e3dad4be87376e500bc (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=3bb44c54e77145c39bccee47a8077319]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:17.279261 24847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c3475bcad94e3dad4be87376e500bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.279278 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c3475bcad94e3dad4be87376e500bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.290134 26928 consensus_queue.cc:1035] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:17.295456 24981 tablet_service.cc:1467] Processing CreateTablet for tablet 754cb0b963534531a95ea9bcbc7d5761 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=946216c79e544a1dab6a99fa3ccbfab3]), partition=
I20240502 01:54:17.295866 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 754cb0b963534531a95ea9bcbc7d5761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.296150 24714 tablet_service.cc:1467] Processing CreateTablet for tablet 754cb0b963534531a95ea9bcbc7d5761 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=946216c79e544a1dab6a99fa3ccbfab3]), partition=
I20240502 01:54:17.296548 24714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 754cb0b963534531a95ea9bcbc7d5761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.301568 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 754cb0b963534531a95ea9bcbc7d5761 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=946216c79e544a1dab6a99fa3ccbfab3]), partition=
I20240502 01:54:17.301976 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 754cb0b963534531a95ea9bcbc7d5761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.316282 27037 tablet_bootstrap.cc:492] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:17.319685 27035 tablet_bootstrap.cc:492] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:17.322419 27035 tablet_bootstrap.cc:654] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.330180 27033 tablet_bootstrap.cc:492] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:17.333750 27033 tablet_bootstrap.cc:654] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.335822 27037 tablet_bootstrap.cc:654] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.342185 27037 tablet_bootstrap.cc:492] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:17.343139 27035 tablet_bootstrap.cc:492] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:17.344677 27037 ts_tablet_manager.cc:1367] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20240502 01:54:17.344996 27035 ts_tablet_manager.cc:1367] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.004s
I20240502 01:54:17.347358 27035 raft_consensus.cc:348] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:17.348676 27033 tablet_bootstrap.cc:492] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:17.349639 27037 raft_consensus.cc:348] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:17.353149 27033 ts_tablet_manager.cc:1367] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20240502 01:54:17.354485 27037 raft_consensus.cc:374] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.354694 27035 raft_consensus.cc:374] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.354995 27035 raft_consensus.cc:729] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.355759 27037 raft_consensus.cc:729] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.356415 27033 raft_consensus.cc:348] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:17.357254 27037 consensus_queue.cc:260] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:17.357805 27035 consensus_queue.cc:260] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:17.359009 27035 ts_tablet_manager.cc:1398] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 01:54:17.359246 27033 raft_consensus.cc:374] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.359772 27037 ts_tablet_manager.cc:1398] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 01:54:17.360028 27033 raft_consensus.cc:729] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.360421 27033 consensus_queue.cc:260] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:17.360965 27035 tablet_bootstrap.cc:492] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:17.362984 27033 ts_tablet_manager.cc:1398] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:54:17.363471 27033 tablet_bootstrap.cc:492] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:17.364809 27035 tablet_bootstrap.cc:654] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.370630 27033 tablet_bootstrap.cc:654] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.379302 27035 tablet_bootstrap.cc:492] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:17.379940 27037 tablet_bootstrap.cc:492] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:17.382730 27037 tablet_bootstrap.cc:654] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.394706 27035 ts_tablet_manager.cc:1367] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20240502 01:54:17.396256 27037 tablet_bootstrap.cc:492] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:17.398303 27037 ts_tablet_manager.cc:1367] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240502 01:54:17.399360 27033 tablet_bootstrap.cc:492] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:17.400314 27037 raft_consensus.cc:348] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:17.405429 27035 raft_consensus.cc:348] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:17.405956 27033 ts_tablet_manager.cc:1367] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20240502 01:54:17.406200 27035 raft_consensus.cc:374] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.407217 27033 raft_consensus.cc:348] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:17.407529 27035 raft_consensus.cc:729] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.408823 27035 consensus_queue.cc:260] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:17.412164 26925 raft_consensus.cc:482] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:17.412447 26925 raft_consensus.cc:504] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:17.414343 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "754cb0b963534531a95ea9bcbc7d5761" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
W20240502 01:54:17.415287 24670 leader_election.cc:341] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:17.415668 26925 leader_election.cc:288] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:17.416064 27037 raft_consensus.cc:374] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.416358 27037 raft_consensus.cc:729] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.412004 26906 raft_consensus.cc:482] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:17.416581 26906 raft_consensus.cc:504] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:17.417536 26906 leader_election.cc:288] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:17.418363 27035 ts_tablet_manager.cc:1398] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240502 01:54:17.418826 27037 consensus_queue.cc:260] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:17.419328 27033 raft_consensus.cc:374] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.425014 27033 raft_consensus.cc:729] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.425945 27033 consensus_queue.cc:260] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:17.428797 27037 ts_tablet_manager.cc:1398] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240502 01:54:17.420781 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee09342ee0f54863a158cb76d2112086" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:17.429661 24867 raft_consensus.cc:2454] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:17.430176 24934 leader_election.cc:302] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:17.430502 26906 raft_consensus.cc:2781] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:17.430636 26906 raft_consensus.cc:482] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:17.430740 26906 raft_consensus.cc:3037] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.431128 27037 tablet_bootstrap.cc:492] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:17.431474 27033 ts_tablet_manager.cc:1398] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240502 01:54:17.433169 27033 tablet_bootstrap.cc:492] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:17.434110 26906 raft_consensus.cc:504] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:17.436439 27037 tablet_bootstrap.cc:654] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.437211 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee09342ee0f54863a158cb76d2112086" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:17.437636 24867 raft_consensus.cc:3037] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.439461 26906 leader_election.cc:288] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:17.440685 24867 raft_consensus.cc:2454] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:17.441207 24934 leader_election.cc:302] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:17.446678 27033 tablet_bootstrap.cc:654] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.452061 26905 raft_consensus.cc:2781] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:17.453564 27037 tablet_bootstrap.cc:492] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:17.454877 27033 tablet_bootstrap.cc:492] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:17.455247 21836 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:17.471343 27037 ts_tablet_manager.cc:1367] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.005s
I20240502 01:54:17.473572 27033 ts_tablet_manager.cc:1367] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.003s
I20240502 01:54:17.477452 27037 raft_consensus.cc:348] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:17.477757 27037 raft_consensus.cc:374] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.479223 27033 raft_consensus.cc:348] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:17.479444 27033 raft_consensus.cc:374] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.479534 27033 raft_consensus.cc:729] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.479826 27033 consensus_queue.cc:260] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:17.477844 27037 raft_consensus.cc:729] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.480506 27037 consensus_queue.cc:260] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:17.487403 27033 ts_tablet_manager.cc:1398] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 01:54:17.488024 27037 ts_tablet_manager.cc:1398] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240502 01:54:17.494442 26906 raft_consensus.cc:482] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:17.494669 26906 raft_consensus.cc:504] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:17.495704 26906 leader_election.cc:288] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:17.496008 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "754cb0b963534531a95ea9bcbc7d5761" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:17.496234 24732 raft_consensus.cc:2454] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:17.497124 24934 leader_election.cc:302] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:17.497411 26906 raft_consensus.cc:2781] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:17.497543 26906 raft_consensus.cc:482] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:17.497661 26906 raft_consensus.cc:3037] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.503417 26906 raft_consensus.cc:504] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:17.504289 26906 leader_election.cc:288] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:17.505127 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "754cb0b963534531a95ea9bcbc7d5761" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:17.505362 24732 raft_consensus.cc:3037] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.505940 27044 tablet_bootstrap.cc:492] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:17.508651 24732 raft_consensus.cc:2454] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:17.510814 27036 tablet_bootstrap.cc:492] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:17.512307 24934 leader_election.cc:302] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:17.513360 27038 tablet_bootstrap.cc:492] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:17.515200 27038 tablet_bootstrap.cc:654] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.527065 27036 tablet_bootstrap.cc:654] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.534698 26906 raft_consensus.cc:2781] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:17.534869 26906 raft_consensus.cc:686] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:54:17.535162 26906 consensus_queue.cc:237] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:17.548791 22117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee09342ee0f54863a158cb76d2112086" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:54:17.549866 22117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee09342ee0f54863a158cb76d2112086" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:54:17.565541 26905 raft_consensus.cc:686] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:54:17.565891 26905 consensus_queue.cc:237] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:17.571218 26537 raft_consensus.cc:482] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:17.571385 26537 raft_consensus.cc:504] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:17.572054 26537 leader_election.cc:288] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:17.572773 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c3475bcad94e3dad4be87376e500bc" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:17.572999 22117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c3475bcad94e3dad4be87376e500bc" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:54:17.573050 24732 raft_consensus.cc:2454] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:17.573132 27044 tablet_bootstrap.cc:654] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
W20240502 01:54:17.573500 24803 leader_election.cc:341] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:17.573714 24802 leader_election.cc:302] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters: c64840657cdf4b93980bd34120c94d9a
I20240502 01:54:17.574155 26537 raft_consensus.cc:2781] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:17.574277 26537 raft_consensus.cc:482] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:17.574373 26537 raft_consensus.cc:3037] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.577454 26537 raft_consensus.cc:504] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:17.578663 26537 leader_election.cc:288] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:17.578920 27038 tablet_bootstrap.cc:492] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:17.578930 22117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c3475bcad94e3dad4be87376e500bc" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:54:17.579206 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c3475bcad94e3dad4be87376e500bc" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:17.579425 24732 raft_consensus.cc:3037] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.579437 27038 ts_tablet_manager.cc:1367] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.000s
W20240502 01:54:17.579429 24803 leader_election.cc:341] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:17.582296 24732 raft_consensus.cc:2454] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:17.582674 24802 leader_election.cc:302] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters: c64840657cdf4b93980bd34120c94d9a
I20240502 01:54:17.583185 26537 raft_consensus.cc:2781] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:17.583745 26537 raft_consensus.cc:686] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:17.584163 27038 raft_consensus.cc:348] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:17.584592 26537 consensus_queue.cc:237] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:17.585364 27038 raft_consensus.cc:374] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.586095 27038 raft_consensus.cc:729] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.587482 27038 consensus_queue.cc:260] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:17.590921 27038 ts_tablet_manager.cc:1398] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:54:17.601327 21709 catalog_manager.cc:5484] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:17.604585 21707 catalog_manager.cc:5484] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:17.607807 27036 tablet_bootstrap.cc:492] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:17.611920 21707 catalog_manager.cc:5484] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:17.624871 26933 raft_consensus.cc:482] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:17.625114 26933 raft_consensus.cc:504] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:17.625955 26933 leader_election.cc:288] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:17.627713 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "754cb0b963534531a95ea9bcbc7d5761" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:17.634709 27044 tablet_bootstrap.cc:492] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:17.634975 27036 ts_tablet_manager.cc:1367] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.124s user 0.005s sys 0.000s
I20240502 01:54:17.635157 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "754cb0b963534531a95ea9bcbc7d5761" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:17.635454 24732 raft_consensus.cc:2379] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c2a28baa7ee2469294d94b7fe5455cff in current term 1: Already voted for candidate 41c70f373c9d49a49891b03a12c6cb2b in this term.
I20240502 01:54:17.636189 22185 leader_election.cc:302] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:17.637593 26926 raft_consensus.cc:3037] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.640581 26926 raft_consensus.cc:2726] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:17.640779 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "754cb0b963534531a95ea9bcbc7d5761" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:17.643082 27044 ts_tablet_manager.cc:1367] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.137s user 0.005s sys 0.000s
I20240502 01:54:17.645044 27044 raft_consensus.cc:348] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:17.645530 27036 raft_consensus.cc:348] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:17.645815 27036 raft_consensus.cc:374] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.645974 27036 raft_consensus.cc:729] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.646293 27036 consensus_queue.cc:260] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:17.646901 24671 leader_election.cc:302] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28157bc3e678428fb5f93c077d528513; no voters: 41c70f373c9d49a49891b03a12c6cb2b, c2a28baa7ee2469294d94b7fe5455cff
I20240502 01:54:17.648666 27031 raft_consensus.cc:2726] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:17.649384 27036 ts_tablet_manager.cc:1398] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:54:17.649590 27044 raft_consensus.cc:374] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.651805 27044 raft_consensus.cc:729] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.662014 27036 tablet_bootstrap.cc:492] T ee09342ee0f54863a158cb76d2112086 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:17.662241 27044 consensus_queue.cc:260] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:17.664646 27036 tablet_bootstrap.cc:654] T ee09342ee0f54863a158cb76d2112086 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
W20240502 01:54:17.669770 24936 leader_election.cc:341] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:54:17.670327 24936 leader_election.cc:341] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:17.671455 27044 ts_tablet_manager.cc:1398] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20240502 01:54:17.682102 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "754cb0b963534531a95ea9bcbc7d5761" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:17.682343 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "754cb0b963534531a95ea9bcbc7d5761" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:17.682420 22250 raft_consensus.cc:2454] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:17.683436 24979 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:17.696920 22231 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:17.719611 22098 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:17.719702 21714 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:17.727694 27036 tablet_bootstrap.cc:492] T ee09342ee0f54863a158cb76d2112086 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:17.728065 27036 ts_tablet_manager.cc:1367] T ee09342ee0f54863a158cb76d2112086 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.012s
I20240502 01:54:17.728746 24732 raft_consensus.cc:1261] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:17.729214 27036 raft_consensus.cc:348] T ee09342ee0f54863a158cb76d2112086 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:17.729674 27036 raft_consensus.cc:374] T ee09342ee0f54863a158cb76d2112086 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.729688 27063 consensus_queue.cc:1035] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:17.729810 27036 raft_consensus.cc:729] T ee09342ee0f54863a158cb76d2112086 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.730230 27036 consensus_queue.cc:260] T ee09342ee0f54863a158cb76d2112086 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:17.730909 27036 ts_tablet_manager.cc:1398] T ee09342ee0f54863a158cb76d2112086 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:17.735826 22250 raft_consensus.cc:1261] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:17.736527 27063 consensus_queue.cc:1035] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:17.801887 27075 tablet.cc:1722] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:17.802552 27075 tablet.cc:1620] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:17.810835 21714 catalog_manager.cc:5576] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375) does not have the latest schema for tablet 754cb0b963534531a95ea9bcbc7d5761 (table rebalancer_dynamic_table [id=946216c79e544a1dab6a99fa3ccbfab3]). Expected version 1 got 0
I20240502 01:54:17.854575 21714 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=946216c79e544a1dab6a99fa3ccbfab3] alter complete (version 1)
I20240502 01:54:17.882201 24848 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:17.883363 27074 tablet.cc:1722] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:17.884078 27074 tablet.cc:1620] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:17.921850 27076 raft_consensus.cc:482] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:17.922036 27076 raft_consensus.cc:504] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:17.922924 27076 leader_election.cc:288] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:17.923415 26537 raft_consensus.cc:482] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:17.923581 26537 raft_consensus.cc:504] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:17.923817 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0e8db8c3414ce0a5ff12200fe6b544" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:17.924108 24732 raft_consensus.cc:2454] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:17.924127 26537 leader_election.cc:288] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:17.924779 24537 leader_election.cc:302] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:54:17.925014 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0e8db8c3414ce0a5ff12200fe6b544" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:17.925231 24602 raft_consensus.cc:2454] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:17.925418 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0e8db8c3414ce0a5ff12200fe6b544" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:17.925608 24867 raft_consensus.cc:2454] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:17.926127 24802 leader_election.cc:302] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:17.926275 27076 raft_consensus.cc:2781] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:17.926371 27076 raft_consensus.cc:482] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:17.926436 27076 raft_consensus.cc:3037] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.927640 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0e8db8c3414ce0a5ff12200fe6b544" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:17.927866 24732 raft_consensus.cc:2454] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:17.929222 26537 raft_consensus.cc:2781] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:17.929323 26537 raft_consensus.cc:482] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:17.929409 26537 raft_consensus.cc:3037] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.930534 27076 raft_consensus.cc:504] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:17.931551 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0e8db8c3414ce0a5ff12200fe6b544" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:17.932096 26537 raft_consensus.cc:504] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:17.932721 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0e8db8c3414ce0a5ff12200fe6b544" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:17.933004 24732 raft_consensus.cc:3037] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.934008 24867 raft_consensus.cc:2379] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0c64d40b5d7644bd8b29feaa80eab72f in current term 1: Already voted for candidate d7449bb1df214bf4a5f93e4631312f08 in this term.
I20240502 01:54:17.935665 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0e8db8c3414ce0a5ff12200fe6b544" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:17.935770 24732 raft_consensus.cc:2454] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:17.935880 24602 raft_consensus.cc:2379] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d7449bb1df214bf4a5f93e4631312f08 in current term 1: Already voted for candidate 0c64d40b5d7644bd8b29feaa80eab72f in this term.
I20240502 01:54:17.936295 24537 leader_election.cc:302] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:54:17.936636 26555 raft_consensus.cc:2781] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:17.936851 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0e8db8c3414ce0a5ff12200fe6b544" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:17.937103 24735 raft_consensus.cc:2379] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d7449bb1df214bf4a5f93e4631312f08 in current term 1: Already voted for candidate 0c64d40b5d7644bd8b29feaa80eab72f in this term.
I20240502 01:54:17.937561 24802 leader_election.cc:302] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7449bb1df214bf4a5f93e4631312f08; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513
I20240502 01:54:17.937853 27061 raft_consensus.cc:2726] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:54:17.939071 27076 leader_election.cc:288] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:17.939447 26537 leader_election.cc:288] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:17.941072 21714 catalog_manager.cc:5576] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093) does not have the latest schema for tablet 754cb0b963534531a95ea9bcbc7d5761 (table rebalancer_dynamic_table [id=946216c79e544a1dab6a99fa3ccbfab3]). Expected version 1 got 0
I20240502 01:54:17.950981 27078 tablet.cc:1722] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:17.953626 21714 catalog_manager.cc:5576] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093) does not have the latest schema for tablet 754cb0b963534531a95ea9bcbc7d5761 (table rebalancer_dynamic_table [id=946216c79e544a1dab6a99fa3ccbfab3]). Expected version 1 got 0
I20240502 01:54:17.957430 27063 consensus_queue.cc:1035] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:17.970281 21707 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:17.975100 22117 raft_consensus.cc:3037] T ee09342ee0f54863a158cb76d2112086 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.978633 27050 consensus_queue.cc:1035] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:17.994030 27075 tablet.cc:1722] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:17.994578 27075 tablet.cc:1620] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:18.012372 27074 tablet.cc:1722] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:18.012965 27074 tablet.cc:1620] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:18.015050 26555 raft_consensus.cc:686] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:54:18.015367 26555 consensus_queue.cc:237] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.017371 21707 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=946216c79e544a1dab6a99fa3ccbfab3] alter complete (version 2)
I20240502 01:54:18.031164 21707 catalog_manager.cc:5484] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:18.033850 27078 tablet.cc:1620] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff: MemRowSet was empty: no flush needed.
I20240502 01:54:18.044492 27078 tablet.cc:1722] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:18.049616 27078 tablet.cc:1620] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff: MemRowSet was empty: no flush needed.
I20240502 01:54:18.053706 21714 catalog_manager.cc:5576] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093) does not have the latest schema for tablet 754cb0b963534531a95ea9bcbc7d5761 (table rebalancer_dynamic_table [id=946216c79e544a1dab6a99fa3ccbfab3]). Expected version 2 got 1
I20240502 01:54:18.099401 21857 consensus_queue.cc:237] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.101588 24735 raft_consensus.cc:1261] T d52f4b5c815f4188a09eb12ec5375361 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:18.101761 22117 raft_consensus.cc:1261] T d52f4b5c815f4188a09eb12ec5375361 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:18.102157 26867 consensus_queue.cc:1035] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:18.102489 27053 consensus_queue.cc:1035] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:18.106626 26867 raft_consensus.cc:2932] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:18.108811 22117 raft_consensus.cc:2932] T d52f4b5c815f4188a09eb12ec5375361 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:18.113363 27047 raft_consensus.cc:482] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:18.113345 21707 catalog_manager.cc:5484] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:18.113569 27047 raft_consensus.cc:504] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:18.114568 27047 leader_election.cc:288] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:18.116101 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c3475bcad94e3dad4be87376e500bc" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:18.116441 24732 raft_consensus.cc:2379] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c64840657cdf4b93980bd34120c94d9a in current term 1: Already voted for candidate d7449bb1df214bf4a5f93e4631312f08 in this term.
I20240502 01:54:18.117144 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c3475bcad94e3dad4be87376e500bc" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:18.119073 22054 leader_election.cc:302] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:18.119360 27047 raft_consensus.cc:3037] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.120172 26537 consensus_queue.cc:1035] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:18.122731 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet d52f4b5c815f4188a09eb12ec5375361 with delete_type TABLET_DATA_TOMBSTONED (TS c2a28baa7ee2469294d94b7fe5455cff not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:18.122839 21707 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:18.123788 27108 tablet_replica.cc:330] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:18.124161 27108 raft_consensus.cc:2229] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:18.124612 27108 raft_consensus.cc:2258] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:18.125563 27108 ts_tablet_manager.cc:1875] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:18.128060 27047 raft_consensus.cc:2726] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:18.139247 27108 ts_tablet_manager.cc:1888] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:54:18.139498 27108 log.cc:1196] T d52f4b5c815f4188a09eb12ec5375361 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/d52f4b5c815f4188a09eb12ec5375361
I20240502 01:54:18.141469 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet d52f4b5c815f4188a09eb12ec5375361 (table rebalance_test_table_17 [id=7a17c6db79004eb696e7dd281fdc4f3b]) successfully deleted
I20240502 01:54:18.144026 27074 tablet.cc:1722] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:18.146620 27061 consensus_queue.cc:1035] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:18.146818 27075 tablet.cc:1722] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:18.149276 24732 raft_consensus.cc:2932] T d52f4b5c815f4188a09eb12ec5375361 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:18.162485 27087 tablet.cc:1722] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:18.167910 21714 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=946216c79e544a1dab6a99fa3ccbfab3] alter complete (version 3)
I20240502 01:54:18.273936 21709 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:18.274232 21709 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:18.278056 21709 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 754cb0b963534531a95ea9bcbc7d5761
I20240502 01:54:18.279203 24713 tablet_service.cc:1514] Processing DeleteTablet for tablet 754cb0b963534531a95ea9bcbc7d5761 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:18 UTC) from {username='slave'} at 127.0.0.1:41650
I20240502 01:54:18.279630 24979 tablet_service.cc:1514] Processing DeleteTablet for tablet 754cb0b963534531a95ea9bcbc7d5761 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:18 UTC) from {username='slave'} at 127.0.0.1:39964
I20240502 01:54:18.279866 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet 754cb0b963534531a95ea9bcbc7d5761 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:18 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:18.280167 27108 tablet_replica.cc:330] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:18.280421 27108 raft_consensus.cc:2229] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:18.280695 27108 raft_consensus.cc:2258] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:18.281728 27108 ts_tablet_manager.cc:1875] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:18.287950 27108 ts_tablet_manager.cc:1888] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:18.288100 27108 log.cc:1196] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/754cb0b963534531a95ea9bcbc7d5761
I20240502 01:54:18.288595 27108 ts_tablet_manager.cc:1909] T 754cb0b963534531a95ea9bcbc7d5761 P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:18.302348 27134 tablet_replica.cc:330] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:54:18.304454 27134 raft_consensus.cc:2229] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:18.304870 27134 raft_consensus.cc:2258] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:18.306267 27135 tablet_replica.cc:330] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:54:18.306505 27135 raft_consensus.cc:2229] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:18.307183 27134 ts_tablet_manager.cc:1875] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:18.307214 27135 raft_consensus.cc:2258] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:18.308001 27135 ts_tablet_manager.cc:1875] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:18.308257 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 754cb0b963534531a95ea9bcbc7d5761 (table rebalancer_dynamic_table_new_name [id=946216c79e544a1dab6a99fa3ccbfab3]) successfully deleted
I20240502 01:54:18.314365 27135 ts_tablet_manager.cc:1888] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:18.314549 27135 log.cc:1196] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/754cb0b963534531a95ea9bcbc7d5761
I20240502 01:54:18.316746 27135 ts_tablet_manager.cc:1909] T 754cb0b963534531a95ea9bcbc7d5761 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting consensus metadata
I20240502 01:54:18.318404 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet 754cb0b963534531a95ea9bcbc7d5761 (table rebalancer_dynamic_table_new_name [id=946216c79e544a1dab6a99fa3ccbfab3]) successfully deleted
I20240502 01:54:18.332476 27134 ts_tablet_manager.cc:1888] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:18.333045 27134 log.cc:1196] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/754cb0b963534531a95ea9bcbc7d5761
I20240502 01:54:18.333700 27134 ts_tablet_manager.cc:1909] T 754cb0b963534531a95ea9bcbc7d5761 P 28157bc3e678428fb5f93c077d528513: Deleting consensus metadata
I20240502 01:54:18.336557 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet 754cb0b963534531a95ea9bcbc7d5761 (table rebalancer_dynamic_table_new_name [id=946216c79e544a1dab6a99fa3ccbfab3]) successfully deleted
I20240502 01:54:18.385079 21710 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:18.409348 24848 tablet_service.cc:1467] Processing CreateTablet for tablet 452496fa41b544cd8f42b69659eac714 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e3ce68924cba47d9b626c1b38e17a72b]), partition=
I20240502 01:54:18.409680 24713 tablet_service.cc:1467] Processing CreateTablet for tablet 452496fa41b544cd8f42b69659eac714 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e3ce68924cba47d9b626c1b38e17a72b]), partition=
I20240502 01:54:18.409799 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 452496fa41b544cd8f42b69659eac714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.410101 24713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 452496fa41b544cd8f42b69659eac714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.410995 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 452496fa41b544cd8f42b69659eac714 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e3ce68924cba47d9b626c1b38e17a72b]), partition=
I20240502 01:54:18.411356 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 452496fa41b544cd8f42b69659eac714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.415951 27146 tablet_bootstrap.cc:492] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:18.418458 27146 tablet_bootstrap.cc:654] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.422426 27149 tablet_bootstrap.cc:492] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:18.424757 27149 tablet_bootstrap.cc:654] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.434798 27149 tablet_bootstrap.cc:492] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:18.435519 27148 tablet_bootstrap.cc:492] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:18.435698 27146 tablet_bootstrap.cc:492] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:18.436014 27146 ts_tablet_manager.cc:1367] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.006s
I20240502 01:54:18.437548 27149 ts_tablet_manager.cc:1367] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20240502 01:54:18.438740 27146 raft_consensus.cc:348] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.439170 27146 raft_consensus.cc:374] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.439354 27146 raft_consensus.cc:729] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.439715 27146 consensus_queue.cc:260] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.439913 27148 tablet_bootstrap.cc:654] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.441186 27149 raft_consensus.cc:348] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.441414 27149 raft_consensus.cc:374] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.441510 27149 raft_consensus.cc:729] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.441756 27149 consensus_queue.cc:260] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.444486 27149 ts_tablet_manager.cc:1398] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 01:54:18.447499 27148 tablet_bootstrap.cc:492] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:18.447666 27148 ts_tablet_manager.cc:1367] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20240502 01:54:18.451647 27148 raft_consensus.cc:348] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.451958 27148 raft_consensus.cc:374] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.452116 27148 raft_consensus.cc:729] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.452447 27148 consensus_queue.cc:260] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.454263 27146 ts_tablet_manager.cc:1398] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240502 01:54:18.457132 27148 ts_tablet_manager.cc:1398] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:54:18.478569 26555 consensus_queue.cc:1035] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:18.488168 26555 consensus_queue.cc:1035] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:18.571343 21707 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:18.589788 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 1962a92af64a4249ab5a041458094936 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=f03cad8416b24f2c8dc87cdc63843ec7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.590354 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1962a92af64a4249ab5a041458094936. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.591377 21967 tablet_service.cc:1467] Processing CreateTablet for tablet ad8bc5be633d4646a7c10d2483ed4e5b (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=f03cad8416b24f2c8dc87cdc63843ec7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.593384 24713 tablet_service.cc:1467] Processing CreateTablet for tablet 1962a92af64a4249ab5a041458094936 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=f03cad8416b24f2c8dc87cdc63843ec7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.593724 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad8bc5be633d4646a7c10d2483ed4e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.593868 24713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1962a92af64a4249ab5a041458094936. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.593868 24848 tablet_service.cc:1467] Processing CreateTablet for tablet c41a6df530ce411db79aa3c582ffc869 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=f03cad8416b24f2c8dc87cdc63843ec7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.594308 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c41a6df530ce411db79aa3c582ffc869. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.597517 22231 tablet_service.cc:1467] Processing CreateTablet for tablet ad8bc5be633d4646a7c10d2483ed4e5b (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=f03cad8416b24f2c8dc87cdc63843ec7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.597556 24715 tablet_service.cc:1467] Processing CreateTablet for tablet c41a6df530ce411db79aa3c582ffc869 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=f03cad8416b24f2c8dc87cdc63843ec7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.597966 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad8bc5be633d4646a7c10d2483ed4e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.598062 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c41a6df530ce411db79aa3c582ffc869. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.621376 27146 tablet_bootstrap.cc:492] T c41a6df530ce411db79aa3c582ffc869 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:18.623685 27146 tablet_bootstrap.cc:654] T c41a6df530ce411db79aa3c582ffc869 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.629150 27149 tablet_bootstrap.cc:492] T c41a6df530ce411db79aa3c582ffc869 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:18.630950 27149 tablet_bootstrap.cc:654] T c41a6df530ce411db79aa3c582ffc869 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.631201 24979 tablet_service.cc:1467] Processing CreateTablet for tablet 1962a92af64a4249ab5a041458094936 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=f03cad8416b24f2c8dc87cdc63843ec7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.631384 24981 tablet_service.cc:1467] Processing CreateTablet for tablet ad8bc5be633d4646a7c10d2483ed4e5b (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=f03cad8416b24f2c8dc87cdc63843ec7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.631794 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1962a92af64a4249ab5a041458094936. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.632016 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad8bc5be633d4646a7c10d2483ed4e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.647055 27149 tablet_bootstrap.cc:492] T c41a6df530ce411db79aa3c582ffc869 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:18.647596 27149 ts_tablet_manager.cc:1367] T c41a6df530ce411db79aa3c582ffc869 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.007s
I20240502 01:54:18.648759 27149 raft_consensus.cc:348] T c41a6df530ce411db79aa3c582ffc869 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.649147 27149 raft_consensus.cc:374] T c41a6df530ce411db79aa3c582ffc869 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.649369 27149 raft_consensus.cc:729] T c41a6df530ce411db79aa3c582ffc869 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.649808 27149 consensus_queue.cc:260] T c41a6df530ce411db79aa3c582ffc869 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.649876 24981 tablet_service.cc:1467] Processing CreateTablet for tablet c41a6df530ce411db79aa3c582ffc869 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=f03cad8416b24f2c8dc87cdc63843ec7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.653684 27149 ts_tablet_manager.cc:1398] T c41a6df530ce411db79aa3c582ffc869 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:18.657266 27146 tablet_bootstrap.cc:492] T c41a6df530ce411db79aa3c582ffc869 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:18.657485 27146 ts_tablet_manager.cc:1367] T c41a6df530ce411db79aa3c582ffc869 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.009s
I20240502 01:54:18.658568 27146 raft_consensus.cc:348] T c41a6df530ce411db79aa3c582ffc869 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.660178 27146 raft_consensus.cc:374] T c41a6df530ce411db79aa3c582ffc869 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.660326 27146 raft_consensus.cc:729] T c41a6df530ce411db79aa3c582ffc869 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.660691 27146 consensus_queue.cc:260] T c41a6df530ce411db79aa3c582ffc869 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.661712 27146 ts_tablet_manager.cc:1398] T c41a6df530ce411db79aa3c582ffc869 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:18.662461 27198 tablet_bootstrap.cc:492] T 1962a92af64a4249ab5a041458094936 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:18.662914 27146 tablet_bootstrap.cc:492] T 1962a92af64a4249ab5a041458094936 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:18.670183 27198 tablet_bootstrap.cc:654] T 1962a92af64a4249ab5a041458094936 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.670462 27146 tablet_bootstrap.cc:654] T 1962a92af64a4249ab5a041458094936 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.659677 27202 tablet_bootstrap.cc:492] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:18.689028 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c41a6df530ce411db79aa3c582ffc869. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.691085 27202 tablet_bootstrap.cc:654] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.696249 27146 tablet_bootstrap.cc:492] T 1962a92af64a4249ab5a041458094936 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:18.697604 27146 ts_tablet_manager.cc:1367] T 1962a92af64a4249ab5a041458094936 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.000s
I20240502 01:54:18.699472 27146 raft_consensus.cc:348] T 1962a92af64a4249ab5a041458094936 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.705276 27146 raft_consensus.cc:374] T 1962a92af64a4249ab5a041458094936 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.705937 27146 raft_consensus.cc:729] T 1962a92af64a4249ab5a041458094936 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.707836 27146 consensus_queue.cc:260] T 1962a92af64a4249ab5a041458094936 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.710058 27200 tablet_bootstrap.cc:492] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:18.711002 27146 ts_tablet_manager.cc:1398] T 1962a92af64a4249ab5a041458094936 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 01:54:18.712800 27202 tablet_bootstrap.cc:492] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:18.713018 27202 ts_tablet_manager.cc:1367] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.010s
I20240502 01:54:18.714143 27202 raft_consensus.cc:348] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.714488 27202 raft_consensus.cc:374] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.714623 27202 raft_consensus.cc:729] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.714915 27202 consensus_queue.cc:260] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.716301 27202 ts_tablet_manager.cc:1398] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:18.716681 27202 tablet_bootstrap.cc:492] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:18.719381 27202 tablet_bootstrap.cc:654] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.732776 27202 tablet_bootstrap.cc:492] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:18.733065 27202 ts_tablet_manager.cc:1367] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
I20240502 01:54:18.734104 27202 raft_consensus.cc:348] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:18.734622 27202 raft_consensus.cc:374] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.734755 27202 raft_consensus.cc:729] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.735123 27202 consensus_queue.cc:260] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:18.736064 27202 ts_tablet_manager.cc:1398] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:18.778788 27113 raft_consensus.cc:482] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:18.779029 27113 raft_consensus.cc:504] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.780004 27113 leader_election.cc:288] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:18.781038 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452496fa41b544cd8f42b69659eac714" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:18.781306 21857 raft_consensus.cc:2454] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:18.781996 24668 leader_election.cc:302] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:18.782816 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452496fa41b544cd8f42b69659eac714" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:18.783077 24867 raft_consensus.cc:2454] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:18.793671 27113 raft_consensus.cc:2781] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:18.794065 27113 raft_consensus.cc:482] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:18.794340 27113 raft_consensus.cc:3037] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.800729 27200 tablet_bootstrap.cc:654] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.805685 27202 tablet_bootstrap.cc:492] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:18.807816 27202 tablet_bootstrap.cc:654] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.808444 27197 tablet_bootstrap.cc:492] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:18.807799 27113 raft_consensus.cc:504] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.810823 27197 tablet_bootstrap.cc:654] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.810714 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452496fa41b544cd8f42b69659eac714" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:18.810986 24867 raft_consensus.cc:3037] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.814062 27198 tablet_bootstrap.cc:492] T 1962a92af64a4249ab5a041458094936 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:18.815014 27198 ts_tablet_manager.cc:1367] T 1962a92af64a4249ab5a041458094936 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.153s user 0.047s sys 0.040s
I20240502 01:54:18.817641 24867 raft_consensus.cc:2454] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:18.818233 27200 tablet_bootstrap.cc:492] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:18.818434 27200 ts_tablet_manager.cc:1367] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.108s user 0.004s sys 0.000s
I20240502 01:54:18.818549 24669 leader_election.cc:302] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:18.818889 27031 raft_consensus.cc:2781] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:18.819444 27200 raft_consensus.cc:348] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:18.819763 27200 raft_consensus.cc:374] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.819878 27200 raft_consensus.cc:729] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.820199 27200 consensus_queue.cc:260] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:18.820442 27198 raft_consensus.cc:348] T 1962a92af64a4249ab5a041458094936 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.820752 27198 raft_consensus.cc:374] T 1962a92af64a4249ab5a041458094936 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.820860 27198 raft_consensus.cc:729] T 1962a92af64a4249ab5a041458094936 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.821996 27198 consensus_queue.cc:260] T 1962a92af64a4249ab5a041458094936 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.823118 27200 ts_tablet_manager.cc:1398] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:18.825788 27198 ts_tablet_manager.cc:1398] T 1962a92af64a4249ab5a041458094936 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240502 01:54:18.829825 27031 raft_consensus.cc:686] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:18.829999 27202 tablet_bootstrap.cc:492] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:18.830197 27202 ts_tablet_manager.cc:1367] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.008s
I20240502 01:54:18.830482 27031 consensus_queue.cc:237] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.831069 27202 raft_consensus.cc:348] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.831326 27202 raft_consensus.cc:374] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.831444 27202 raft_consensus.cc:729] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.831702 27202 consensus_queue.cc:260] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.832294 27202 ts_tablet_manager.cc:1398] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:18.844311 27206 raft_consensus.cc:482] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:18.844486 27206 raft_consensus.cc:504] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:18.845701 27206 leader_election.cc:288] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:18.846405 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8bc5be633d4646a7c10d2483ed4e5b" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
W20240502 01:54:18.847144 22184 leader_election.cc:341] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:18.850214 26905 raft_consensus.cc:482] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:18.850395 26905 raft_consensus.cc:504] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.851145 26905 leader_election.cc:288] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:18.852146 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1962a92af64a4249ab5a041458094936" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:18.852241 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1962a92af64a4249ab5a041458094936" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:18.852406 24602 raft_consensus.cc:2454] T 1962a92af64a4249ab5a041458094936 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:18.852494 24732 raft_consensus.cc:2454] T 1962a92af64a4249ab5a041458094936 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:18.853154 24934 leader_election.cc:302] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:18.853824 26905 raft_consensus.cc:2781] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:18.853948 26905 raft_consensus.cc:482] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:18.854036 26905 raft_consensus.cc:3037] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.859563 26905 raft_consensus.cc:504] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.861222 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1962a92af64a4249ab5a041458094936" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:18.861516 24602 raft_consensus.cc:3037] T 1962a92af64a4249ab5a041458094936 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.861672 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1962a92af64a4249ab5a041458094936" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:18.861943 24732 raft_consensus.cc:3037] T 1962a92af64a4249ab5a041458094936 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.867210 24732 raft_consensus.cc:2454] T 1962a92af64a4249ab5a041458094936 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:18.867704 24934 leader_election.cc:302] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:18.873313 26905 leader_election.cc:288] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:18.873551 26905 raft_consensus.cc:2781] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:18.873694 26905 raft_consensus.cc:686] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:54:18.873719 24602 raft_consensus.cc:2454] T 1962a92af64a4249ab5a041458094936 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:18.874018 26905 consensus_queue.cc:237] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.889897 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452496fa41b544cd8f42b69659eac714" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:18.890175 21857 raft_consensus.cc:3037] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.894119 21857 raft_consensus.cc:2454] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:18.904002 27113 leader_election.cc:288] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
W20240502 01:54:18.940774 24934 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.199:35737, user_credentials={real_user=slave}} blocked reactor thread for 70931.2us
I20240502 01:54:18.943840 26905 raft_consensus.cc:482] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:18.944181 26905 raft_consensus.cc:504] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.945533 26905 leader_election.cc:288] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:18.946717 21710 catalog_manager.cc:5484] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:18.948693 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a6df530ce411db79aa3c582ffc869" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:18.949131 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8bc5be633d4646a7c10d2483ed4e5b" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:18.949229 24867 raft_consensus.cc:2454] T c41a6df530ce411db79aa3c582ffc869 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:18.949360 25000 raft_consensus.cc:2454] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:54:18.955462 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a6df530ce411db79aa3c582ffc869" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:18.955919 24732 raft_consensus.cc:2454] T c41a6df530ce411db79aa3c582ffc869 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:18.957561 24934 leader_election.cc:302] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:18.957643 22187 leader_election.cc:302] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, c2a28baa7ee2469294d94b7fe5455cff; no voters: 06543b05a16f42099bc86db2a0284412
I20240502 01:54:18.958215 27206 raft_consensus.cc:2781] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:18.958345 27206 raft_consensus.cc:482] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:18.958417 27206 raft_consensus.cc:3037] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.959033 27216 raft_consensus.cc:2781] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:18.959235 27216 raft_consensus.cc:482] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:18.959369 27216 raft_consensus.cc:3037] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.961457 27206 raft_consensus.cc:504] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:18.962905 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8bc5be633d4646a7c10d2483ed4e5b" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
W20240502 01:54:18.963656 22184 leader_election.cc:341] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:18.965963 27206 leader_election.cc:288] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:18.967195 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8bc5be633d4646a7c10d2483ed4e5b" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:18.967473 25000 raft_consensus.cc:3037] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.968801 27216 raft_consensus.cc:504] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.970667 25000 raft_consensus.cc:2454] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:54:18.972088 27197 tablet_bootstrap.cc:492] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:18.972311 27197 ts_tablet_manager.cc:1367] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.164s user 0.009s sys 0.006s
I20240502 01:54:18.973335 27197 raft_consensus.cc:348] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:18.973584 27197 raft_consensus.cc:374] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.973711 27197 raft_consensus.cc:729] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.974098 27197 consensus_queue.cc:260] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:18.974975 27197 ts_tablet_manager.cc:1398] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:18.980086 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a6df530ce411db79aa3c582ffc869" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:18.980381 24867 raft_consensus.cc:3037] T c41a6df530ce411db79aa3c582ffc869 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.984337 24867 raft_consensus.cc:2454] T c41a6df530ce411db79aa3c582ffc869 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:18.985248 24934 leader_election.cc:302] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:18.985836 27223 raft_consensus.cc:2781] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:18.986130 27223 raft_consensus.cc:686] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:54:18.986590 27223 consensus_queue.cc:237] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:18.988781 27216 leader_election.cc:288] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:18.989655 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a6df530ce411db79aa3c582ffc869" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:18.989953 24732 raft_consensus.cc:3037] T c41a6df530ce411db79aa3c582ffc869 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.988552 21710 catalog_manager.cc:5484] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:18.991497 22187 leader_election.cc:302] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, c2a28baa7ee2469294d94b7fe5455cff; no voters: 06543b05a16f42099bc86db2a0284412
I20240502 01:54:18.991853 27206 raft_consensus.cc:2781] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:18.991992 27206 raft_consensus.cc:686] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:54:18.992311 27206 consensus_queue.cc:237] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:19.002207 24732 raft_consensus.cc:2454] T c41a6df530ce411db79aa3c582ffc869 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:19.015419 21707 catalog_manager.cc:5484] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:19.026414 21707 catalog_manager.cc:5484] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:19.100982 21967 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:19.103660 24583 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:19.137856 24715 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:19.138386 21836 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:19.142019 21710 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:19.145921 24981 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:19.208607 21857 raft_consensus.cc:1261] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:19.208899 24867 raft_consensus.cc:1261] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:19.209324 27031 consensus_queue.cc:1035] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:19.209609 27215 consensus_queue.cc:1035] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:19.230768 22231 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:19.247804 24848 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:19.268477 27251 tablet.cc:1722] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:19.269382 21710 catalog_manager.cc:5576] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737) does not have the latest schema for tablet 452496fa41b544cd8f42b69659eac714 (table rebalancer_dynamic_table [id=e3ce68924cba47d9b626c1b38e17a72b]). Expected version 1 got 0
I20240502 01:54:19.271883 27251 tablet.cc:1620] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:19.298823 21710 catalog_manager.cc:5576] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737) does not have the latest schema for tablet 452496fa41b544cd8f42b69659eac714 (table rebalancer_dynamic_table [id=e3ce68924cba47d9b626c1b38e17a72b]). Expected version 1 got 0
I20240502 01:54:19.314858 21707 catalog_manager.cc:5576] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785) does not have the latest schema for tablet 452496fa41b544cd8f42b69659eac714 (table rebalancer_dynamic_table [id=e3ce68924cba47d9b626c1b38e17a72b]). Expected version 1 got 0
I20240502 01:54:19.329255 21710 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=e3ce68924cba47d9b626c1b38e17a72b] alter complete (version 1)
I20240502 01:54:19.332244 27252 tablet.cc:1722] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:19.333130 27252 tablet.cc:1620] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:19.333659 27216 consensus_queue.cc:1035] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:19.338510 22098 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:19.366053 21707 catalog_manager.cc:5576] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785) does not have the latest schema for tablet 452496fa41b544cd8f42b69659eac714 (table rebalancer_dynamic_table [id=e3ce68924cba47d9b626c1b38e17a72b]). Expected version 1 got 0
I20240502 01:54:19.363406 27255 tablet.cc:1722] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:19.367611 27255 tablet.cc:1620] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77: MemRowSet was empty: no flush needed.
I20240502 01:54:19.392246 27216 consensus_queue.cc:1035] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:19.439698 21710 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:19.444006 27225 raft_consensus.cc:482] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:19.444204 27225 raft_consensus.cc:504] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:19.445798 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8bc5be633d4646a7c10d2483ed4e5b" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:19.446087 24999 raft_consensus.cc:2379] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 06543b05a16f42099bc86db2a0284412 in current term 1: Already voted for candidate c2a28baa7ee2469294d94b7fe5455cff in this term.
I20240502 01:54:19.448758 27251 tablet.cc:1722] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:19.449250 27225 leader_election.cc:288] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:19.449280 27251 tablet.cc:1620] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:19.450986 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8bc5be633d4646a7c10d2483ed4e5b" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:19.451501 27252 tablet.cc:1722] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:19.452006 27252 tablet.cc:1620] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:19.453282 21924 leader_election.cc:302] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06543b05a16f42099bc86db2a0284412; no voters: 41c70f373c9d49a49891b03a12c6cb2b, c2a28baa7ee2469294d94b7fe5455cff
I20240502 01:54:19.454989 27255 tablet.cc:1722] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:19.455451 27255 tablet.cc:1620] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77: MemRowSet was empty: no flush needed.
I20240502 01:54:19.466003 27225 raft_consensus.cc:3037] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.470036 27225 raft_consensus.cc:2726] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:19.468959 21707 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=e3ce68924cba47d9b626c1b38e17a72b] alter complete (version 2)
I20240502 01:54:19.491320 27206 consensus_queue.cc:1035] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:19.517236 27206 consensus_queue.cc:1035] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:19.536726 22117 consensus_queue.cc:237] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:19.538908 22250 raft_consensus.cc:1261] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:19.539417 24999 raft_consensus.cc:1261] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:19.539454 27051 consensus_queue.cc:1035] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:19.539963 27051 consensus_queue.cc:1035] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:19.542179 27207 raft_consensus.cc:2932] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:19.542704 22250 raft_consensus.cc:2932] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:19.548202 21710 catalog_manager.cc:5484] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:19.546411 24999 raft_consensus.cc:2932] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:19.552600 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b924192d8da4a8bb7df2e6c5e4cbd4f with delete_type TABLET_DATA_TOMBSTONED (TS 759257ea7bef415ca698e60d82199d77 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:19.565387 27287 tablet_replica.cc:330] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:19.565665 27287 raft_consensus.cc:2229] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:19.565953 27287 raft_consensus.cc:2258] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:19.566606 27287 ts_tablet_manager.cc:1875] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:19.572551 27287 ts_tablet_manager.cc:1888] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:54:19.572726 27287 log.cc:1196] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/4b924192d8da4a8bb7df2e6c5e4cbd4f
I20240502 01:54:19.574016 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 4b924192d8da4a8bb7df2e6c5e4cbd4f (table rebalance_test_table_9 [id=714c5d94b5b64cb29dc398713b1727fe]) successfully deleted
I20240502 01:54:19.574959 21707 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:19.583451 27255 tablet.cc:1722] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:19.583420 27251 tablet.cc:1722] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:19.585819 27252 tablet.cc:1722] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:19.593111 21707 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=e3ce68924cba47d9b626c1b38e17a72b] alter complete (version 3)
I20240502 01:54:19.599855 21707 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:19.610029 27216 consensus_queue.cc:1035] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:19.610841 22098 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb2e5d929334665ae41e33056c520bc (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=e6f275d6ff034b158e43e06075db0828]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.610841 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb2e5d929334665ae41e33056c520bc (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=e6f275d6ff034b158e43e06075db0828]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.611303 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb2e5d929334665ae41e33056c520bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.611303 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb2e5d929334665ae41e33056c520bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.612102 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb2e5d929334665ae41e33056c520bc (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=e6f275d6ff034b158e43e06075db0828]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.612547 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb2e5d929334665ae41e33056c520bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.612565 24848 tablet_service.cc:1467] Processing CreateTablet for tablet f0622f8af0e045a4a3578b030bebdf4f (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=e6f275d6ff034b158e43e06075db0828]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.612995 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0622f8af0e045a4a3578b030bebdf4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.616067 27216 consensus_queue.cc:1035] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:19.619576 24981 tablet_service.cc:1467] Processing CreateTablet for tablet f0622f8af0e045a4a3578b030bebdf4f (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=e6f275d6ff034b158e43e06075db0828]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.620061 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0622f8af0e045a4a3578b030bebdf4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.621377 24846 tablet_service.cc:1467] Processing CreateTablet for tablet c351a8e007d04327bb14303d7b32d5be (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=e6f275d6ff034b158e43e06075db0828]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.621816 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c351a8e007d04327bb14303d7b32d5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.624583 22231 tablet_service.cc:1467] Processing CreateTablet for tablet f0622f8af0e045a4a3578b030bebdf4f (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=e6f275d6ff034b158e43e06075db0828]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.625041 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0622f8af0e045a4a3578b030bebdf4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.625980 24582 tablet_service.cc:1467] Processing CreateTablet for tablet c351a8e007d04327bb14303d7b32d5be (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=e6f275d6ff034b158e43e06075db0828]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.626394 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c351a8e007d04327bb14303d7b32d5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.629710 22100 tablet_service.cc:1467] Processing CreateTablet for tablet c351a8e007d04327bb14303d7b32d5be (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=e6f275d6ff034b158e43e06075db0828]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.630172 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c351a8e007d04327bb14303d7b32d5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.636960 27299 tablet_bootstrap.cc:492] T 0fb2e5d929334665ae41e33056c520bc P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:19.641434 27299 tablet_bootstrap.cc:654] T 0fb2e5d929334665ae41e33056c520bc P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.653223 27302 tablet_bootstrap.cc:492] T f0622f8af0e045a4a3578b030bebdf4f P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:19.660003 27303 tablet_bootstrap.cc:492] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:19.659052 27305 tablet_bootstrap.cc:492] T f0622f8af0e045a4a3578b030bebdf4f P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:19.663183 27301 tablet_bootstrap.cc:492] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:19.666124 27303 tablet_bootstrap.cc:654] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.666211 27301 tablet_bootstrap.cc:654] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.666486 27302 tablet_bootstrap.cc:654] T f0622f8af0e045a4a3578b030bebdf4f P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.671195 27305 tablet_bootstrap.cc:654] T f0622f8af0e045a4a3578b030bebdf4f P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.672304 27301 tablet_bootstrap.cc:492] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:19.673380 27301 ts_tablet_manager.cc:1367] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 01:54:19.674991 27300 tablet_bootstrap.cc:492] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:19.675863 27301 raft_consensus.cc:348] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.676259 27301 raft_consensus.cc:374] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.676412 27301 raft_consensus.cc:729] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.676973 27299 tablet_bootstrap.cc:492] T 0fb2e5d929334665ae41e33056c520bc P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:19.676856 27301 consensus_queue.cc:260] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.677176 27299 ts_tablet_manager.cc:1367] T 0fb2e5d929334665ae41e33056c520bc P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.041s user 0.011s sys 0.010s
I20240502 01:54:19.677551 27303 tablet_bootstrap.cc:492] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:19.677722 27303 ts_tablet_manager.cc:1367] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.007s
I20240502 01:54:19.678151 27299 raft_consensus.cc:348] T 0fb2e5d929334665ae41e33056c520bc P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.678471 27299 raft_consensus.cc:374] T 0fb2e5d929334665ae41e33056c520bc P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.678625 27299 raft_consensus.cc:729] T 0fb2e5d929334665ae41e33056c520bc P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.678593 27303 raft_consensus.cc:348] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.678834 27303 raft_consensus.cc:374] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.679040 27303 raft_consensus.cc:729] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.679092 27300 tablet_bootstrap.cc:654] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.679473 27299 consensus_queue.cc:260] T 0fb2e5d929334665ae41e33056c520bc P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.680296 27299 ts_tablet_manager.cc:1398] T 0fb2e5d929334665ae41e33056c520bc P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:19.680684 27303 consensus_queue.cc:260] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.681380 27303 ts_tablet_manager.cc:1398] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:54:19.682873 27303 tablet_bootstrap.cc:492] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:19.683233 27301 ts_tablet_manager.cc:1398] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 01:54:19.684582 27303 tablet_bootstrap.cc:654] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.685763 27305 tablet_bootstrap.cc:492] T f0622f8af0e045a4a3578b030bebdf4f P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:19.685802 27254 raft_consensus.cc:482] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:19.685958 27305 ts_tablet_manager.cc:1367] T f0622f8af0e045a4a3578b030bebdf4f P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240502 01:54:19.685953 27254 raft_consensus.cc:504] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.686900 27254 leader_election.cc:288] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:19.686978 27305 raft_consensus.cc:348] T f0622f8af0e045a4a3578b030bebdf4f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:19.687227 27305 raft_consensus.cc:374] T f0622f8af0e045a4a3578b030bebdf4f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.687335 27305 raft_consensus.cc:729] T f0622f8af0e045a4a3578b030bebdf4f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.687588 22117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb2e5d929334665ae41e33056c520bc" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:54:19.687600 27305 consensus_queue.cc:260] T f0622f8af0e045a4a3578b030bebdf4f P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:19.688323 27305 ts_tablet_manager.cc:1398] T f0622f8af0e045a4a3578b030bebdf4f P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240502 01:54:19.688455 24538 leader_election.cc:341] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:19.689842 27303 tablet_bootstrap.cc:492] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:19.692049 27300 tablet_bootstrap.cc:492] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:19.694283 27303 ts_tablet_manager.cc:1367] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.006s
I20240502 01:54:19.694561 27300 ts_tablet_manager.cc:1367] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240502 01:54:19.695430 27303 raft_consensus.cc:348] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.695644 27303 raft_consensus.cc:374] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.696507 27300 raft_consensus.cc:348] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:19.696791 27303 raft_consensus.cc:729] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.696971 27300 raft_consensus.cc:374] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.697294 27303 consensus_queue.cc:260] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.697583 27300 raft_consensus.cc:729] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.698042 27300 consensus_queue.cc:260] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:19.698812 27303 ts_tablet_manager.cc:1398] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 01:54:19.700522 27300 ts_tablet_manager.cc:1398] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:19.703400 27301 tablet_bootstrap.cc:492] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:19.703905 27051 raft_consensus.cc:482] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:19.704057 27051 raft_consensus.cc:504] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.705731 27301 tablet_bootstrap.cc:654] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.706377 27302 tablet_bootstrap.cc:492] T f0622f8af0e045a4a3578b030bebdf4f P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:19.706596 27302 ts_tablet_manager.cc:1367] T f0622f8af0e045a4a3578b030bebdf4f P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.003s
I20240502 01:54:19.707015 27051 leader_election.cc:288] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:19.707803 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb2e5d929334665ae41e33056c520bc" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:19.708060 21854 raft_consensus.cc:2454] T 0fb2e5d929334665ae41e33056c520bc P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:54:19.708580 22053 leader_election.cc:302] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:54:19.709128 27051 raft_consensus.cc:2781] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:19.709260 27051 raft_consensus.cc:482] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:19.709355 27051 raft_consensus.cc:3037] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.710254 21710 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:19.710481 21710 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:19.716104 21710 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 452496fa41b544cd8f42b69659eac714
I20240502 01:54:19.716012 27051 raft_consensus.cc:504] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.716852 24848 tablet_service.cc:1514] Processing DeleteTablet for tablet 452496fa41b544cd8f42b69659eac714 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:19 UTC) from {username='slave'} at 127.0.0.1:51592
I20240502 01:54:19.717274 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb2e5d929334665ae41e33056c520bc" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:19.717640 21854 raft_consensus.cc:3037] T 0fb2e5d929334665ae41e33056c520bc P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.719271 27315 tablet_replica.cc:330] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08: stopping tablet replica
I20240502 01:54:19.719553 27315 raft_consensus.cc:2229] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:19.719871 27315 raft_consensus.cc:2258] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:19.720350 24715 tablet_service.cc:1514] Processing DeleteTablet for tablet 452496fa41b544cd8f42b69659eac714 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:19 UTC) from {username='slave'} at 127.0.0.1:41650
I20240502 01:54:19.720723 21854 raft_consensus.cc:2454] T 0fb2e5d929334665ae41e33056c520bc P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:54:19.721153 27315 ts_tablet_manager.cc:1875] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:19.721321 27051 leader_election.cc:288] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:19.723574 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 452496fa41b544cd8f42b69659eac714 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:19 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:19.723908 27287 tablet_replica.cc:330] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:19.724108 27287 raft_consensus.cc:2229] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:19.724323 27287 raft_consensus.cc:2258] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:19.725564 27287 ts_tablet_manager.cc:1875] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:19.726967 22053 leader_election.cc:302] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:54:19.728168 27051 raft_consensus.cc:2781] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:19.728195 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb2e5d929334665ae41e33056c520bc" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:19.728343 27315 ts_tablet_manager.cc:1888] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:19.728394 27051 raft_consensus.cc:686] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:54:19.728467 27315 log.cc:1196] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/wals/452496fa41b544cd8f42b69659eac714
I20240502 01:54:19.728454 24603 raft_consensus.cc:3037] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.728701 27051 consensus_queue.cc:237] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.729019 27315 ts_tablet_manager.cc:1909] T 452496fa41b544cd8f42b69659eac714 P d7449bb1df214bf4a5f93e4631312f08: Deleting consensus metadata
I20240502 01:54:19.730067 21700 catalog_manager.cc:4830] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): tablet 452496fa41b544cd8f42b69659eac714 (table rebalancer_dynamic_table_new_name [id=e3ce68924cba47d9b626c1b38e17a72b]) successfully deleted
I20240502 01:54:19.730003 27302 raft_consensus.cc:348] T f0622f8af0e045a4a3578b030bebdf4f P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:19.730216 27302 raft_consensus.cc:374] T f0622f8af0e045a4a3578b030bebdf4f P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.730326 27302 raft_consensus.cc:729] T f0622f8af0e045a4a3578b030bebdf4f P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.730603 27302 consensus_queue.cc:260] T f0622f8af0e045a4a3578b030bebdf4f P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:19.731313 27302 ts_tablet_manager.cc:1398] T f0622f8af0e045a4a3578b030bebdf4f P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20240502 01:54:19.731293 24603 raft_consensus.cc:2454] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:54:19.728214 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb2e5d929334665ae41e33056c520bc" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:19.731925 24602 raft_consensus.cc:2362] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1. Re-sending same reply.
I20240502 01:54:19.732074 27207 raft_consensus.cc:482] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:19.732259 27207 raft_consensus.cc:504] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.732977 27207 leader_election.cc:288] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:19.734226 27302 tablet_bootstrap.cc:492] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:19.734330 24866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c351a8e007d04327bb14303d7b32d5be" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
W20240502 01:54:19.735134 22054 leader_election.cc:341] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:19.735127 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c351a8e007d04327bb14303d7b32d5be" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
W20240502 01:54:19.735845 22054 leader_election.cc:341] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:19.736032 22054 leader_election.cc:302] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:19.736591 27207 raft_consensus.cc:2726] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:19.740576 27287 ts_tablet_manager.cc:1888] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:19.741892 27302 tablet_bootstrap.cc:654] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.745878 27287 log.cc:1196] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/452496fa41b544cd8f42b69659eac714
I20240502 01:54:19.746724 27287 ts_tablet_manager.cc:1909] T 452496fa41b544cd8f42b69659eac714 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:19.747098 27301 tablet_bootstrap.cc:492] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:19.747327 27301 ts_tablet_manager.cc:1367] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.006s
I20240502 01:54:19.748473 27301 raft_consensus.cc:348] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.748791 27301 raft_consensus.cc:374] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.748950 27301 raft_consensus.cc:729] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.749346 27301 consensus_queue.cc:260] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.749459 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 452496fa41b544cd8f42b69659eac714 (table rebalancer_dynamic_table_new_name [id=e3ce68924cba47d9b626c1b38e17a72b]) successfully deleted
I20240502 01:54:19.752130 27301 ts_tablet_manager.cc:1398] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:54:19.750408 21710 catalog_manager.cc:5484] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:19.755169 27316 tablet_replica.cc:330] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:54:19.755472 27316 raft_consensus.cc:2229] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:19.755828 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb2e5d929334665ae41e33056c520bc" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:19.755971 27302 tablet_bootstrap.cc:492] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:19.755939 27316 raft_consensus.cc:2258] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:19.756099 21854 raft_consensus.cc:2379] T 0fb2e5d929334665ae41e33056c520bc P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c64d40b5d7644bd8b29feaa80eab72f in current term 1: Already voted for candidate c64840657cdf4b93980bd34120c94d9a in this term.
I20240502 01:54:19.756160 27302 ts_tablet_manager.cc:1367] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.010s
I20240502 01:54:19.756857 24536 leader_election.cc:302] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a
I20240502 01:54:19.757148 27316 ts_tablet_manager.cc:1875] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:19.756986 27302 raft_consensus.cc:348] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.757269 27302 raft_consensus.cc:374] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.757364 27302 raft_consensus.cc:729] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.757594 27302 consensus_queue.cc:260] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.758345 27302 ts_tablet_manager.cc:1398] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:54:19.758456 27254 raft_consensus.cc:2726] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:19.768790 27316 ts_tablet_manager.cc:1888] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:19.769053 27316 log.cc:1196] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/452496fa41b544cd8f42b69659eac714
I20240502 01:54:19.769587 27316 ts_tablet_manager.cc:1909] T 452496fa41b544cd8f42b69659eac714 P 28157bc3e678428fb5f93c077d528513: Deleting consensus metadata
I20240502 01:54:19.770687 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet 452496fa41b544cd8f42b69659eac714 (table rebalancer_dynamic_table_new_name [id=e3ce68924cba47d9b626c1b38e17a72b]) successfully deleted
I20240502 01:54:19.804436 26537 raft_consensus.cc:482] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:19.804622 26537 raft_consensus.cc:504] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.805454 26537 leader_election.cc:288] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:19.805910 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c351a8e007d04327bb14303d7b32d5be" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:54:19.806246 22118 raft_consensus.cc:2454] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:19.806712 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c351a8e007d04327bb14303d7b32d5be" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:19.806965 24602 raft_consensus.cc:2454] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:19.807641 24802 leader_election.cc:302] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:19.808838 26537 raft_consensus.cc:2781] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:19.808979 26537 raft_consensus.cc:482] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:19.809067 26537 raft_consensus.cc:3037] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.811154 26537 raft_consensus.cc:504] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.812039 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c351a8e007d04327bb14303d7b32d5be" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:54:19.812283 22118 raft_consensus.cc:3037] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.812794 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c351a8e007d04327bb14303d7b32d5be" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:19.813006 24602 raft_consensus.cc:3037] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.815717 24602 raft_consensus.cc:2454] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:19.816319 26537 leader_election.cc:288] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:19.816812 24802 leader_election.cc:302] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:19.817165 22118 raft_consensus.cc:2454] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:19.817301 26537 raft_consensus.cc:2781] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:19.817446 26537 raft_consensus.cc:686] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:19.817749 26537 consensus_queue.cc:237] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.823182 21710 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:19.841989 24981 tablet_service.cc:1467] Processing CreateTablet for tablet feee459a6a9242ba8329c09b77c4017b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0d4511aeb5714acdab13730700d24b41]), partition=
I20240502 01:54:19.842447 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feee459a6a9242ba8329c09b77c4017b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.844256 24848 tablet_service.cc:1467] Processing CreateTablet for tablet feee459a6a9242ba8329c09b77c4017b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0d4511aeb5714acdab13730700d24b41]), partition=
I20240502 01:54:19.844640 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feee459a6a9242ba8329c09b77c4017b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.845451 24583 tablet_service.cc:1467] Processing CreateTablet for tablet feee459a6a9242ba8329c09b77c4017b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0d4511aeb5714acdab13730700d24b41]), partition=
I20240502 01:54:19.845821 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feee459a6a9242ba8329c09b77c4017b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.851608 27302 tablet_bootstrap.cc:492] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:19.854131 27302 tablet_bootstrap.cc:654] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.879550 21710 catalog_manager.cc:5484] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:19.886595 27300 tablet_bootstrap.cc:492] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:19.886865 27301 tablet_bootstrap.cc:492] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:19.888419 27301 tablet_bootstrap.cc:654] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.888762 27300 tablet_bootstrap.cc:654] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.891561 27300 tablet_bootstrap.cc:492] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:19.891829 27300 ts_tablet_manager.cc:1367] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 01:54:19.892896 27300 raft_consensus.cc:348] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.893281 27300 raft_consensus.cc:374] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.893417 27300 raft_consensus.cc:729] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.893791 27300 consensus_queue.cc:260] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.894627 27300 ts_tablet_manager.cc:1398] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:19.906453 27301 tablet_bootstrap.cc:492] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:19.906638 27301 ts_tablet_manager.cc:1367] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.001s
I20240502 01:54:19.907732 27301 raft_consensus.cc:348] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.907954 27301 raft_consensus.cc:374] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.908059 27301 raft_consensus.cc:729] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.908282 27301 consensus_queue.cc:260] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.910724 27301 ts_tablet_manager.cc:1398] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:19.923172 27302 tablet_bootstrap.cc:492] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:19.923347 27302 ts_tablet_manager.cc:1367] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.072s user 0.005s sys 0.004s
I20240502 01:54:19.924288 27302 raft_consensus.cc:348] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.924512 27302 raft_consensus.cc:374] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.924638 27302 raft_consensus.cc:729] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.925006 27302 consensus_queue.cc:260] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.925717 27302 ts_tablet_manager.cc:1398] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:19.953783 27061 raft_consensus.cc:482] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:19.953955 27061 raft_consensus.cc:504] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.954902 27061 leader_election.cc:288] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:19.955446 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feee459a6a9242ba8329c09b77c4017b" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:19.955722 24602 raft_consensus.cc:2454] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:19.955710 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feee459a6a9242ba8329c09b77c4017b" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:19.955925 24999 raft_consensus.cc:2454] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:19.956379 24804 leader_election.cc:302] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:19.956673 27061 raft_consensus.cc:2781] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:19.956781 27061 raft_consensus.cc:482] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:19.956868 27061 raft_consensus.cc:3037] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.959709 27061 raft_consensus.cc:504] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.966141 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feee459a6a9242ba8329c09b77c4017b" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:19.966348 24999 raft_consensus.cc:3037] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.969337 24999 raft_consensus.cc:2454] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:19.969807 24804 leader_election.cc:302] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:19.970103 26537 raft_consensus.cc:2781] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:19.970266 26537 raft_consensus.cc:686] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:19.970531 26537 consensus_queue.cc:237] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:19.972088 27061 leader_election.cc:288] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:19.973186 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feee459a6a9242ba8329c09b77c4017b" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:19.973429 24602 raft_consensus.cc:3037] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.974812 21710 catalog_manager.cc:5484] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:19.976249 24602 raft_consensus.cc:2454] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:20.001518 27223 raft_consensus.cc:482] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:20.001709 27223 raft_consensus.cc:504] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:20.002424 27223 leader_election.cc:288] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:20.003223 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0622f8af0e045a4a3578b030bebdf4f" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:20.003510 22249 raft_consensus.cc:2454] T f0622f8af0e045a4a3578b030bebdf4f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:20.003916 24936 leader_election.cc:302] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:20.004400 27223 raft_consensus.cc:2781] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:20.004493 27223 raft_consensus.cc:482] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:20.004593 27223 raft_consensus.cc:3037] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.004627 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0622f8af0e045a4a3578b030bebdf4f" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:20.005566 24868 raft_consensus.cc:2454] T f0622f8af0e045a4a3578b030bebdf4f P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:20.009439 27223 raft_consensus.cc:504] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:20.011691 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0622f8af0e045a4a3578b030bebdf4f" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:20.011619 27223 leader_election.cc:288] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:20.011958 22249 raft_consensus.cc:3037] T f0622f8af0e045a4a3578b030bebdf4f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.015329 22249 raft_consensus.cc:2454] T f0622f8af0e045a4a3578b030bebdf4f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:20.015781 24936 leader_election.cc:302] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:20.016130 27223 raft_consensus.cc:2781] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:20.016294 27223 raft_consensus.cc:686] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:54:20.016594 27223 consensus_queue.cc:237] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:20.010627 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0622f8af0e045a4a3578b030bebdf4f" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:20.017464 24868 raft_consensus.cc:3037] T f0622f8af0e045a4a3578b030bebdf4f P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.025307 24868 raft_consensus.cc:2454] T f0622f8af0e045a4a3578b030bebdf4f P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:20.035712 21710 catalog_manager.cc:5484] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:20.084425 21709 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:20.092120 24999 raft_consensus.cc:1261] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.092384 24602 raft_consensus.cc:1261] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.093174 27335 consensus_queue.cc:1035] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:20.094452 27335 consensus_queue.cc:1035] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:20.104328 21709 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet feee459a6a9242ba8329c09b77c4017b (table rebalancer_dynamic_table [id=0d4511aeb5714acdab13730700d24b41]). Expected version 1 got 0
I20240502 01:54:20.104599 27358 tablet.cc:1722] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:20.107221 27358 tablet.cc:1620] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:20.114763 21714 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=0d4511aeb5714acdab13730700d24b41] alter complete (version 1)
I20240502 01:54:20.142310 27370 tablet.cc:1722] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:20.142575 21714 catalog_manager.cc:5576] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677) does not have the latest schema for tablet feee459a6a9242ba8329c09b77c4017b (table rebalancer_dynamic_table [id=0d4511aeb5714acdab13730700d24b41]). Expected version 1 got 0
I20240502 01:54:20.144757 27370 tablet.cc:1620] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:20.144984 27357 tablet.cc:1722] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:20.146072 27357 tablet.cc:1620] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:20.146700 21709 catalog_manager.cc:5576] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677) does not have the latest schema for tablet feee459a6a9242ba8329c09b77c4017b (table rebalancer_dynamic_table [id=0d4511aeb5714acdab13730700d24b41]). Expected version 1 got 0
I20240502 01:54:20.174007 21967 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:20.202769 21836 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:20.203253 24715 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:20.204562 24583 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:20.216284 21714 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:20.229703 24981 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:20.244890 27370 tablet.cc:1722] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:20.245538 27370 tablet.cc:1620] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:20.259758 27358 tablet.cc:1722] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:20.260326 27358 tablet.cc:1620] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:20.266824 22231 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:20.276291 27357 tablet.cc:1722] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:20.276857 27357 tablet.cc:1620] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:20.277068 21714 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=0d4511aeb5714acdab13730700d24b41] alter complete (version 2)
I20240502 01:54:20.279955 27051 consensus_queue.cc:1035] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:20.314244 27061 consensus_queue.cc:1035] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:20.345955 24848 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:20.346122 27061 consensus_queue.cc:1035] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:20.359704 27207 consensus_queue.cc:1035] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:20.391044 21714 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:20.399745 27357 tablet.cc:1722] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:20.400646 27358 tablet.cc:1722] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:20.405385 27370 tablet.cc:1722] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:20.417002 22100 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:20.417547 21714 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=0d4511aeb5714acdab13730700d24b41] alter complete (version 3)
I20240502 01:54:20.440398 27223 consensus_queue.cc:1035] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:20.483645 27216 consensus_queue.cc:1035] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:20.522521 21709 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:20.522784 21709 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:20.526659 21709 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet feee459a6a9242ba8329c09b77c4017b
I20240502 01:54:20.528306 24981 tablet_service.cc:1514] Processing DeleteTablet for tablet feee459a6a9242ba8329c09b77c4017b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:20 UTC) from {username='slave'} at 127.0.0.1:39964
I20240502 01:54:20.543634 24848 tablet_service.cc:1514] Processing DeleteTablet for tablet feee459a6a9242ba8329c09b77c4017b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:20 UTC) from {username='slave'} at 127.0.0.1:51592
I20240502 01:54:20.555191 24583 tablet_service.cc:1514] Processing DeleteTablet for tablet feee459a6a9242ba8329c09b77c4017b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:20 UTC) from {username='slave'} at 127.0.0.1:53832
I20240502 01:54:20.599694 21709 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:20.604449 27394 tablet_replica.cc:330] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:54:20.605434 27397 tablet_replica.cc:330] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:54:20.605765 27397 raft_consensus.cc:2229] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:20.606151 27397 raft_consensus.cc:2258] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:20.608476 27396 tablet_replica.cc:330] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08: stopping tablet replica
I20240502 01:54:20.608803 27396 raft_consensus.cc:2229] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:20.608832 27397 ts_tablet_manager.cc:1875] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:20.609293 27396 raft_consensus.cc:2258] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:20.611315 27394 raft_consensus.cc:2229] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:20.611689 27394 raft_consensus.cc:2258] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:20.617834 27396 ts_tablet_manager.cc:1875] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:20.621749 27397 ts_tablet_manager.cc:1888] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:20.621889 27397 log.cc:1196] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/feee459a6a9242ba8329c09b77c4017b
I20240502 01:54:20.625308 27397 ts_tablet_manager.cc:1909] T feee459a6a9242ba8329c09b77c4017b P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting consensus metadata
I20240502 01:54:20.630717 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet feee459a6a9242ba8329c09b77c4017b (table rebalancer_dynamic_table_new_name [id=0d4511aeb5714acdab13730700d24b41]) successfully deleted
I20240502 01:54:20.632649 24848 tablet_service.cc:1467] Processing CreateTablet for tablet 0a1b190978514cb49efb2b97cc97a65f (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=eda35ad172274afa81b14d7170413fac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.633369 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a1b190978514cb49efb2b97cc97a65f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.635329 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 0a1b190978514cb49efb2b97cc97a65f (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=eda35ad172274afa81b14d7170413fac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.635793 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a1b190978514cb49efb2b97cc97a65f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.641898 27396 ts_tablet_manager.cc:1888] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:20.642112 27396 log.cc:1196] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/wals/feee459a6a9242ba8329c09b77c4017b
I20240502 01:54:20.642462 24846 tablet_service.cc:1467] Processing CreateTablet for tablet ac80dc6d73db4352b9864392badc16c1 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=eda35ad172274afa81b14d7170413fac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.642796 27396 ts_tablet_manager.cc:1909] T feee459a6a9242ba8329c09b77c4017b P d7449bb1df214bf4a5f93e4631312f08: Deleting consensus metadata
I20240502 01:54:20.642865 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac80dc6d73db4352b9864392badc16c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.644225 21700 catalog_manager.cc:4830] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): tablet feee459a6a9242ba8329c09b77c4017b (table rebalancer_dynamic_table_new_name [id=0d4511aeb5714acdab13730700d24b41]) successfully deleted
I20240502 01:54:20.644614 27403 tablet_bootstrap.cc:492] T 0a1b190978514cb49efb2b97cc97a65f P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:20.646589 21967 tablet_service.cc:1467] Processing CreateTablet for tablet ac80dc6d73db4352b9864392badc16c1 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=eda35ad172274afa81b14d7170413fac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.647046 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac80dc6d73db4352b9864392badc16c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.648373 27403 tablet_bootstrap.cc:654] T 0a1b190978514cb49efb2b97cc97a65f P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.648520 24981 tablet_service.cc:1467] Processing CreateTablet for tablet ac80dc6d73db4352b9864392badc16c1 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=eda35ad172274afa81b14d7170413fac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.648995 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac80dc6d73db4352b9864392badc16c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.656392 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 54bb2c6e63de4207a86894acad10cfe3 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=eda35ad172274afa81b14d7170413fac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.656839 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54bb2c6e63de4207a86894acad10cfe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.657079 24582 tablet_service.cc:1467] Processing CreateTablet for tablet 54bb2c6e63de4207a86894acad10cfe3 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=eda35ad172274afa81b14d7170413fac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.657393 27403 tablet_bootstrap.cc:492] T 0a1b190978514cb49efb2b97cc97a65f P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:20.657482 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54bb2c6e63de4207a86894acad10cfe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.657676 27403 ts_tablet_manager.cc:1367] T 0a1b190978514cb49efb2b97cc97a65f P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240502 01:54:20.658878 27403 raft_consensus.cc:348] T 0a1b190978514cb49efb2b97cc97a65f P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:20.659128 27403 raft_consensus.cc:374] T 0a1b190978514cb49efb2b97cc97a65f P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.659227 27403 raft_consensus.cc:729] T 0a1b190978514cb49efb2b97cc97a65f P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.659518 27403 consensus_queue.cc:260] T 0a1b190978514cb49efb2b97cc97a65f P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:20.664963 27403 ts_tablet_manager.cc:1398] T 0a1b190978514cb49efb2b97cc97a65f P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:54:20.669189 27403 tablet_bootstrap.cc:492] T ac80dc6d73db4352b9864392badc16c1 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:20.670325 27405 tablet_bootstrap.cc:492] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:20.671718 27403 tablet_bootstrap.cc:654] T ac80dc6d73db4352b9864392badc16c1 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.673174 27405 tablet_bootstrap.cc:654] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.673732 27409 tablet_bootstrap.cc:492] T ac80dc6d73db4352b9864392badc16c1 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:20.673645 21714 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:20.675702 27409 tablet_bootstrap.cc:654] T ac80dc6d73db4352b9864392badc16c1 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.680361 27405 tablet_bootstrap.cc:492] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:20.680568 27405 ts_tablet_manager.cc:1367] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20240502 01:54:20.681660 27405 raft_consensus.cc:348] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:20.681959 27405 raft_consensus.cc:374] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.682089 27405 raft_consensus.cc:729] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.685498 27405 consensus_queue.cc:260] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:20.686944 27405 ts_tablet_manager.cc:1398] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240502 01:54:20.689371 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 0a1b190978514cb49efb2b97cc97a65f (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=eda35ad172274afa81b14d7170413fac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.689919 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a1b190978514cb49efb2b97cc97a65f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.692221 27409 tablet_bootstrap.cc:492] T ac80dc6d73db4352b9864392badc16c1 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:20.692443 27409 ts_tablet_manager.cc:1367] T ac80dc6d73db4352b9864392badc16c1 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20240502 01:54:20.694666 27409 raft_consensus.cc:348] T ac80dc6d73db4352b9864392badc16c1 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:20.694950 27409 raft_consensus.cc:374] T ac80dc6d73db4352b9864392badc16c1 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.695099 27409 raft_consensus.cc:729] T ac80dc6d73db4352b9864392badc16c1 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.695477 27403 tablet_bootstrap.cc:492] T ac80dc6d73db4352b9864392badc16c1 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:20.695721 27403 ts_tablet_manager.cc:1367] T ac80dc6d73db4352b9864392badc16c1 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.015s
I20240502 01:54:20.695441 27409 consensus_queue.cc:260] T ac80dc6d73db4352b9864392badc16c1 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:20.696743 27403 raft_consensus.cc:348] T ac80dc6d73db4352b9864392badc16c1 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:20.697618 27403 raft_consensus.cc:374] T ac80dc6d73db4352b9864392badc16c1 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.697728 27403 raft_consensus.cc:729] T ac80dc6d73db4352b9864392badc16c1 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.698004 24846 tablet_service.cc:1467] Processing CreateTablet for tablet 54bb2c6e63de4207a86894acad10cfe3 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=eda35ad172274afa81b14d7170413fac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.698004 27403 consensus_queue.cc:260] T ac80dc6d73db4352b9864392badc16c1 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:20.698478 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54bb2c6e63de4207a86894acad10cfe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.698871 27403 ts_tablet_manager.cc:1398] T ac80dc6d73db4352b9864392badc16c1 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:54:20.703680 27412 tablet_bootstrap.cc:492] T 54bb2c6e63de4207a86894acad10cfe3 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:20.706190 27409 ts_tablet_manager.cc:1398] T ac80dc6d73db4352b9864392badc16c1 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 01:54:20.707456 27403 tablet_bootstrap.cc:492] T 54bb2c6e63de4207a86894acad10cfe3 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:20.711205 24846 tablet_service.cc:1467] Processing CreateTablet for tablet eeb8d3ee3a4e4633bb5356d57a174658 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=047edadf279840d28157991051cc9035]), partition=
I20240502 01:54:20.711624 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeb8d3ee3a4e4633bb5356d57a174658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.712566 24713 tablet_service.cc:1467] Processing CreateTablet for tablet eeb8d3ee3a4e4633bb5356d57a174658 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=047edadf279840d28157991051cc9035]), partition=
I20240502 01:54:20.712707 24582 tablet_service.cc:1467] Processing CreateTablet for tablet eeb8d3ee3a4e4633bb5356d57a174658 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=047edadf279840d28157991051cc9035]), partition=
I20240502 01:54:20.712985 24713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeb8d3ee3a4e4633bb5356d57a174658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.713040 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeb8d3ee3a4e4633bb5356d57a174658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.720119 27403 tablet_bootstrap.cc:654] T 54bb2c6e63de4207a86894acad10cfe3 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.720716 27394 ts_tablet_manager.cc:1875] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:20.724146 27412 tablet_bootstrap.cc:654] T 54bb2c6e63de4207a86894acad10cfe3 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.728535 27403 tablet_bootstrap.cc:492] T 54bb2c6e63de4207a86894acad10cfe3 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:20.728871 27403 ts_tablet_manager.cc:1367] T 54bb2c6e63de4207a86894acad10cfe3 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.003s
I20240502 01:54:20.730556 27407 tablet_bootstrap.cc:492] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:20.730474 27403 raft_consensus.cc:348] T 54bb2c6e63de4207a86894acad10cfe3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:20.730811 27403 raft_consensus.cc:374] T 54bb2c6e63de4207a86894acad10cfe3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.732918 27403 raft_consensus.cc:729] T 54bb2c6e63de4207a86894acad10cfe3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.733662 27403 consensus_queue.cc:260] T 54bb2c6e63de4207a86894acad10cfe3 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:20.736275 27403 ts_tablet_manager.cc:1398] T 54bb2c6e63de4207a86894acad10cfe3 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20240502 01:54:20.736634 27403 tablet_bootstrap.cc:492] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:20.742507 27412 tablet_bootstrap.cc:492] T 54bb2c6e63de4207a86894acad10cfe3 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:20.742769 27412 ts_tablet_manager.cc:1367] T 54bb2c6e63de4207a86894acad10cfe3 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.000s
I20240502 01:54:20.743870 27412 raft_consensus.cc:348] T 54bb2c6e63de4207a86894acad10cfe3 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:20.744230 27412 raft_consensus.cc:374] T 54bb2c6e63de4207a86894acad10cfe3 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.744547 27412 raft_consensus.cc:729] T 54bb2c6e63de4207a86894acad10cfe3 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.745033 27412 consensus_queue.cc:260] T 54bb2c6e63de4207a86894acad10cfe3 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:20.746032 27407 tablet_bootstrap.cc:654] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.746074 27412 ts_tablet_manager.cc:1398] T 54bb2c6e63de4207a86894acad10cfe3 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:20.746565 27403 tablet_bootstrap.cc:654] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.751915 27412 tablet_bootstrap.cc:492] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:20.754211 27412 tablet_bootstrap.cc:654] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.760267 27412 tablet_bootstrap.cc:492] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:20.760587 27412 ts_tablet_manager.cc:1367] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20240502 01:54:20.761714 27412 raft_consensus.cc:348] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:20.762229 27412 raft_consensus.cc:374] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.762363 27412 raft_consensus.cc:729] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.762729 27412 consensus_queue.cc:260] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:20.763695 27406 tablet_bootstrap.cc:492] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:20.765080 27406 tablet_bootstrap.cc:654] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.767210 27407 tablet_bootstrap.cc:492] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:20.767791 27407 ts_tablet_manager.cc:1367] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.000s
I20240502 01:54:20.767786 27394 ts_tablet_manager.cc:1888] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:20.767911 27394 log.cc:1196] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/feee459a6a9242ba8329c09b77c4017b
I20240502 01:54:20.768404 27394 ts_tablet_manager.cc:1909] T feee459a6a9242ba8329c09b77c4017b P 41c70f373c9d49a49891b03a12c6cb2b: Deleting consensus metadata
I20240502 01:54:20.769109 27407 raft_consensus.cc:348] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:20.769369 27407 raft_consensus.cc:374] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.769560 27407 raft_consensus.cc:729] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.769904 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet feee459a6a9242ba8329c09b77c4017b (table rebalancer_dynamic_table_new_name [id=0d4511aeb5714acdab13730700d24b41]) successfully deleted
I20240502 01:54:20.771175 27412 ts_tablet_manager.cc:1398] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:54:20.769907 27407 consensus_queue.cc:260] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:20.771822 27403 tablet_bootstrap.cc:492] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:20.772013 27403 ts_tablet_manager.cc:1367] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.002s
I20240502 01:54:20.772433 27407 ts_tablet_manager.cc:1398] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:20.772753 27407 tablet_bootstrap.cc:492] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:20.772799 27403 raft_consensus.cc:348] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:20.773067 27403 raft_consensus.cc:374] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.773160 27403 raft_consensus.cc:729] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.773387 27403 consensus_queue.cc:260] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:20.773947 27403 ts_tablet_manager.cc:1398] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:20.775718 27406 tablet_bootstrap.cc:492] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:20.775924 27406 ts_tablet_manager.cc:1367] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 01:54:20.776787 27406 raft_consensus.cc:348] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:20.777153 27407 tablet_bootstrap.cc:654] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.778425 27412 tablet_bootstrap.cc:492] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:20.778465 27406 raft_consensus.cc:374] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.778635 27406 raft_consensus.cc:729] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.778872 27406 consensus_queue.cc:260] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:20.779537 27406 ts_tablet_manager.cc:1398] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:20.780180 27412 tablet_bootstrap.cc:654] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.787406 27407 tablet_bootstrap.cc:492] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:20.787600 27407 ts_tablet_manager.cc:1367] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.004s
I20240502 01:54:20.788357 27407 raft_consensus.cc:348] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:20.788518 27407 raft_consensus.cc:374] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.788599 27407 raft_consensus.cc:729] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.788823 27407 consensus_queue.cc:260] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:20.789443 27407 ts_tablet_manager.cc:1398] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 01:54:20.796303 27412 tablet_bootstrap.cc:492] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:20.796449 27412 ts_tablet_manager.cc:1367] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.001s
I20240502 01:54:20.797549 27412 raft_consensus.cc:348] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:20.798198 27412 raft_consensus.cc:374] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.798570 27412 raft_consensus.cc:729] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.799314 27412 consensus_queue.cc:260] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:20.800169 27412 ts_tablet_manager.cc:1398] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 01:54:20.812357 27254 raft_consensus.cc:482] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:20.812567 27254 raft_consensus.cc:504] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:20.813688 27254 leader_election.cc:288] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:20.814386 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb8d3ee3a4e4633bb5356d57a174658" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:20.814646 24868 raft_consensus.cc:2454] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:20.815029 24734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb8d3ee3a4e4633bb5356d57a174658" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:20.815246 24734 raft_consensus.cc:2454] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:20.815853 24537 leader_election.cc:302] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:20.816742 27254 raft_consensus.cc:2781] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:20.816967 27254 raft_consensus.cc:482] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:20.817092 27254 raft_consensus.cc:3037] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.820865 27254 raft_consensus.cc:504] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:20.837100 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb8d3ee3a4e4633bb5356d57a174658" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:20.837393 24868 raft_consensus.cc:3037] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.841046 27389 raft_consensus.cc:482] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:20.841274 27389 raft_consensus.cc:504] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:20.842439 27389 leader_election.cc:288] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:20.844594 24866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a1b190978514cb49efb2b97cc97a65f" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:20.844929 24866 raft_consensus.cc:2454] T 0a1b190978514cb49efb2b97cc97a65f P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:20.847558 24669 leader_election.cc:302] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:20.848147 27389 raft_consensus.cc:2781] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:20.848268 27389 raft_consensus.cc:482] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:20.848238 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a1b190978514cb49efb2b97cc97a65f" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:20.848378 27389 raft_consensus.cc:3037] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.848474 24603 raft_consensus.cc:2454] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:20.851390 24868 raft_consensus.cc:2454] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:20.853334 24537 leader_election.cc:302] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:20.853708 27336 raft_consensus.cc:2781] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:20.854189 27336 raft_consensus.cc:686] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:54:20.854559 27336 consensus_queue.cc:237] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:20.857230 24734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb8d3ee3a4e4633bb5356d57a174658" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:20.857422 24734 raft_consensus.cc:3037] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.856370 27389 raft_consensus.cc:504] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:20.863668 27389 leader_election.cc:288] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:20.863838 27254 leader_election.cc:288] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:20.867271 21714 catalog_manager.cc:5484] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:20.870651 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a1b190978514cb49efb2b97cc97a65f" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:20.870847 24603 raft_consensus.cc:3037] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.874586 24866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a1b190978514cb49efb2b97cc97a65f" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:20.874837 24866 raft_consensus.cc:3037] T 0a1b190978514cb49efb2b97cc97a65f P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.875644 24603 raft_consensus.cc:2454] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:20.876470 24669 leader_election.cc:302] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:54:20.877043 27260 raft_consensus.cc:2781] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:20.877197 27260 raft_consensus.cc:686] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:20.877509 27260 consensus_queue.cc:237] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:20.880613 24866 raft_consensus.cc:2454] T 0a1b190978514cb49efb2b97cc97a65f P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:20.881383 24734 raft_consensus.cc:2454] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:20.896750 21709 catalog_manager.cc:5484] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:20.902161 27393 raft_consensus.cc:482] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:20.902410 27393 raft_consensus.cc:504] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:20.903429 27393 leader_election.cc:288] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:20.908658 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac80dc6d73db4352b9864392badc16c1" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:20.912281 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac80dc6d73db4352b9864392badc16c1" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:20.912636 24868 raft_consensus.cc:2454] T ac80dc6d73db4352b9864392badc16c1 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:20.914446 21923 leader_election.cc:302] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:20.915128 27393 raft_consensus.cc:2781] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:20.915275 27393 raft_consensus.cc:482] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:20.915378 27393 raft_consensus.cc:3037] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.916337 25000 raft_consensus.cc:2454] T ac80dc6d73db4352b9864392badc16c1 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:20.925634 27393 raft_consensus.cc:504] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:20.926426 27393 leader_election.cc:288] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:20.926833 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac80dc6d73db4352b9864392badc16c1" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:20.926944 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac80dc6d73db4352b9864392badc16c1" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:20.927074 24868 raft_consensus.cc:3037] T ac80dc6d73db4352b9864392badc16c1 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.927151 25000 raft_consensus.cc:3037] T ac80dc6d73db4352b9864392badc16c1 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.930080 25000 raft_consensus.cc:2454] T ac80dc6d73db4352b9864392badc16c1 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:20.930752 21925 leader_election.cc:302] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:20.931051 26786 raft_consensus.cc:2781] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:20.931193 26786 raft_consensus.cc:686] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:54:20.931442 26786 consensus_queue.cc:237] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:20.931787 24868 raft_consensus.cc:2454] T ac80dc6d73db4352b9864392badc16c1 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:20.948419 21709 catalog_manager.cc:5484] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:20.953922 25000 consensus_queue.cc:237] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:20.955915 21986 raft_consensus.cc:1261] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:20.955974 21857 raft_consensus.cc:1261] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:20.956466 27216 consensus_queue.cc:1035] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:20.956777 27223 consensus_queue.cc:1035] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:20.959921 27223 raft_consensus.cc:2932] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:20.960487 21857 raft_consensus.cc:2932] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:20.961593 21986 raft_consensus.cc:2932] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:20.965768 21709 catalog_manager.cc:5484] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:20.971580 24846 tablet_service.cc:1514] Processing DeleteTablet for tablet 992544bbea6547de96298fd32d74b232 with delete_type TABLET_DATA_TOMBSTONED (TS d7449bb1df214bf4a5f93e4631312f08 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51592
I20240502 01:54:20.972041 27396 tablet_replica.cc:330] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08: stopping tablet replica
I20240502 01:54:20.972306 27396 raft_consensus.cc:2229] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:20.972580 27396 raft_consensus.cc:2258] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:20.973367 27396 ts_tablet_manager.cc:1875] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:20.981094 21709 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:20.983934 27396 ts_tablet_manager.cc:1888] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:54:20.984228 27396 log.cc:1196] T 992544bbea6547de96298fd32d74b232 P d7449bb1df214bf4a5f93e4631312f08: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/wals/992544bbea6547de96298fd32d74b232
I20240502 01:54:20.985572 21700 catalog_manager.cc:4830] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): tablet 992544bbea6547de96298fd32d74b232 (table rebalancer_extra_table_4 [id=c1a2048ad81b49729677603cbe361089]) successfully deleted
I20240502 01:54:20.988170 24734 raft_consensus.cc:1261] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.988350 24868 raft_consensus.cc:1261] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.988790 27254 consensus_queue.cc:1035] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:20.989055 27336 consensus_queue.cc:1035] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:20.993263 21709 catalog_manager.cc:5576] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737) does not have the latest schema for tablet eeb8d3ee3a4e4633bb5356d57a174658 (table rebalancer_dynamic_table [id=047edadf279840d28157991051cc9035]). Expected version 1 got 0
I20240502 01:54:20.994138 27449 tablet.cc:1722] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:20.994931 27449 tablet.cc:1620] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:20.995656 21709 catalog_manager.cc:5576] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375) does not have the latest schema for tablet eeb8d3ee3a4e4633bb5356d57a174658 (table rebalancer_dynamic_table [id=047edadf279840d28157991051cc9035]). Expected version 1 got 0
I20240502 01:54:20.997969 27451 tablet.cc:1722] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:20.998817 27451 tablet.cc:1620] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:21.005483 27452 tablet.cc:1722] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:21.006093 27452 tablet.cc:1620] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:21.006865 21709 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=047edadf279840d28157991051cc9035] alter complete (version 1)
I20240502 01:54:21.008133 21713 catalog_manager.cc:5576] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737) does not have the latest schema for tablet eeb8d3ee3a4e4633bb5356d57a174658 (table rebalancer_dynamic_table [id=047edadf279840d28157991051cc9035]). Expected version 1 got 0
I20240502 01:54:21.118418 21714 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:21.131108 27449 tablet.cc:1722] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:21.131628 27449 tablet.cc:1620] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:21.133100 27452 tablet.cc:1722] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:21.133589 27452 tablet.cc:1620] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:21.140652 27451 tablet.cc:1722] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:21.141182 27451 tablet.cc:1620] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:21.155874 21707 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=047edadf279840d28157991051cc9035] alter complete (version 2)
I20240502 01:54:21.177470 27275 raft_consensus.cc:482] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:21.177652 27275 raft_consensus.cc:504] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:21.178586 27275 leader_election.cc:288] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:21.179909 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bb2c6e63de4207a86894acad10cfe3" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:21.180114 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bb2c6e63de4207a86894acad10cfe3" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:21.180202 24603 raft_consensus.cc:2454] T 54bb2c6e63de4207a86894acad10cfe3 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:54:21.180366 24868 raft_consensus.cc:2454] T 54bb2c6e63de4207a86894acad10cfe3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:54:21.181609 22185 leader_election.cc:302] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:21.181900 27275 raft_consensus.cc:2781] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:21.182013 27275 raft_consensus.cc:482] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:21.182102 27275 raft_consensus.cc:3037] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:21.184203 27275 raft_consensus.cc:504] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:21.184762 27275 leader_election.cc:288] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:21.185109 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bb2c6e63de4207a86894acad10cfe3" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:21.185334 24603 raft_consensus.cc:3037] T 54bb2c6e63de4207a86894acad10cfe3 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:21.185287 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bb2c6e63de4207a86894acad10cfe3" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:21.185528 24868 raft_consensus.cc:3037] T 54bb2c6e63de4207a86894acad10cfe3 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:21.188215 24603 raft_consensus.cc:2454] T 54bb2c6e63de4207a86894acad10cfe3 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:54:21.188248 24868 raft_consensus.cc:2454] T 54bb2c6e63de4207a86894acad10cfe3 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:54:21.188611 22185 leader_election.cc:302] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:21.188969 27275 raft_consensus.cc:2781] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:21.189582 27275 raft_consensus.cc:686] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:54:21.189980 27275 consensus_queue.cc:237] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:21.194056 21714 catalog_manager.cc:5484] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:21.234625 21967 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:21.267756 21714 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:21.282126 27389 consensus_queue.cc:1035] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:21.284606 27449 tablet.cc:1722] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:21.286032 27452 tablet.cc:1722] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:21.286659 27451 tablet.cc:1722] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:21.301437 21712 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=047edadf279840d28157991051cc9035] alter complete (version 3)
I20240502 01:54:21.319213 24981 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:21.321763 27393 consensus_queue.cc:1035] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:21.323766 21836 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:21.351644 27389 consensus_queue.cc:1035] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:21.389760 24715 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:21.390139 22231 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:54:21.391535 22100 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:21.399449 24846 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:21.403836 21714 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:21.404143 21714 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:21.408218 21714 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet eeb8d3ee3a4e4633bb5356d57a174658
I20240502 01:54:21.409595 24582 tablet_service.cc:1514] Processing DeleteTablet for tablet eeb8d3ee3a4e4633bb5356d57a174658 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:21 UTC) from {username='slave'} at 127.0.0.1:53832
I20240502 01:54:21.410105 24846 tablet_service.cc:1514] Processing DeleteTablet for tablet eeb8d3ee3a4e4633bb5356d57a174658 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:21 UTC) from {username='slave'} at 127.0.0.1:51592
I20240502 01:54:21.410385 27396 tablet_replica.cc:330] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08: stopping tablet replica
I20240502 01:54:21.410617 27396 raft_consensus.cc:2229] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:21.410861 27396 raft_consensus.cc:2258] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:21.414115 27396 ts_tablet_manager.cc:1875] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:21.423930 24715 tablet_service.cc:1514] Processing DeleteTablet for tablet eeb8d3ee3a4e4633bb5356d57a174658 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:21 UTC) from {username='slave'} at 127.0.0.1:41650
I20240502 01:54:21.426038 27396 ts_tablet_manager.cc:1888] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:21.426235 27396 log.cc:1196] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/wals/eeb8d3ee3a4e4633bb5356d57a174658
I20240502 01:54:21.426733 27396 ts_tablet_manager.cc:1909] T eeb8d3ee3a4e4633bb5356d57a174658 P d7449bb1df214bf4a5f93e4631312f08: Deleting consensus metadata
I20240502 01:54:21.428737 21700 catalog_manager.cc:4830] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): tablet eeb8d3ee3a4e4633bb5356d57a174658 (table rebalancer_dynamic_table_new_name [id=047edadf279840d28157991051cc9035]) successfully deleted
I20240502 01:54:21.430786 27510 tablet_replica.cc:330] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:54:21.431583 27510 raft_consensus.cc:2229] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:21.432026 27510 raft_consensus.cc:2258] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:21.432997 27510 ts_tablet_manager.cc:1875] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:21.453032 24582 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:21.458971 27510 ts_tablet_manager.cc:1888] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:21.459280 27510 log.cc:1196] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/eeb8d3ee3a4e4633bb5356d57a174658
I20240502 01:54:21.459932 27510 ts_tablet_manager.cc:1909] T eeb8d3ee3a4e4633bb5356d57a174658 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting consensus metadata
I20240502 01:54:21.461691 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet eeb8d3ee3a4e4633bb5356d57a174658 (table rebalancer_dynamic_table_new_name [id=047edadf279840d28157991051cc9035]) successfully deleted
I20240502 01:54:21.505364 27511 tablet_replica.cc:330] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:54:21.507400 27511 raft_consensus.cc:2229] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:21.508467 27511 raft_consensus.cc:2258] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:21.517053 21714 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240502 01:54:21.521625 24803 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.196:43093, user_credentials={real_user=slave}} blocked reactor thread for 75757.3us
I20240502 01:54:21.522640 27511 ts_tablet_manager.cc:1875] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:21.529310 21967 tablet_service.cc:1467] Processing CreateTablet for tablet a1cda7268286458c8eaccbc225784520 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f84e1086c99a45d396af709a16ace85f]), partition=
I20240502 01:54:21.529520 22231 tablet_service.cc:1467] Processing CreateTablet for tablet a1cda7268286458c8eaccbc225784520 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f84e1086c99a45d396af709a16ace85f]), partition=
I20240502 01:54:21.529728 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1cda7268286458c8eaccbc225784520. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:21.529932 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1cda7268286458c8eaccbc225784520. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:21.533461 21836 tablet_service.cc:1467] Processing CreateTablet for tablet a1cda7268286458c8eaccbc225784520 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f84e1086c99a45d396af709a16ace85f]), partition=
I20240502 01:54:21.533831 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1cda7268286458c8eaccbc225784520. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:21.543802 27521 tablet_bootstrap.cc:492] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:21.546595 27521 tablet_bootstrap.cc:654] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:21.549517 27522 tablet_bootstrap.cc:492] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:21.551235 27522 tablet_bootstrap.cc:654] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:21.559813 27522 tablet_bootstrap.cc:492] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:21.560472 27522 ts_tablet_manager.cc:1367] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 01:54:21.561499 27511 ts_tablet_manager.cc:1888] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:21.561777 27511 log.cc:1196] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/eeb8d3ee3a4e4633bb5356d57a174658
I20240502 01:54:21.561897 27522 raft_consensus.cc:348] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:21.564043 27521 tablet_bootstrap.cc:492] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:21.564262 27521 ts_tablet_manager.cc:1367] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.008s
I20240502 01:54:21.565238 27521 raft_consensus.cc:348] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:21.565529 27521 raft_consensus.cc:374] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:21.565725 27521 raft_consensus.cc:729] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:21.566102 27521 consensus_queue.cc:260] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:21.566987 27521 ts_tablet_manager.cc:1398] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:21.567294 27522 raft_consensus.cc:374] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:21.567476 27522 raft_consensus.cc:729] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:21.568670 27522 consensus_queue.cc:260] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:21.569763 27522 ts_tablet_manager.cc:1398] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 01:54:21.572862 27525 tablet_bootstrap.cc:492] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:21.589464 27511 ts_tablet_manager.cc:1909] T eeb8d3ee3a4e4633bb5356d57a174658 P 28157bc3e678428fb5f93c077d528513: Deleting consensus metadata
I20240502 01:54:21.594317 27525 tablet_bootstrap.cc:654] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:21.594257 27393 consensus_queue.cc:1035] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:21.594507 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet eeb8d3ee3a4e4633bb5356d57a174658 (table rebalancer_dynamic_table_new_name [id=047edadf279840d28157991051cc9035]) successfully deleted
I20240502 01:54:21.608814 27525 tablet_bootstrap.cc:492] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:21.609004 27525 ts_tablet_manager.cc:1367] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.002s
I20240502 01:54:21.609753 27525 raft_consensus.cc:348] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:21.609948 27525 raft_consensus.cc:374] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:21.612039 27525 raft_consensus.cc:729] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:21.612546 27525 consensus_queue.cc:260] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:21.644033 27525 ts_tablet_manager.cc:1398] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20240502 01:54:21.710245 27275 consensus_queue.cc:1035] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:21.718305 27393 raft_consensus.cc:482] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:21.718498 27393 raft_consensus.cc:504] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:21.719254 27393 leader_election.cc:288] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:21.719949 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cda7268286458c8eaccbc225784520" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:21.720065 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cda7268286458c8eaccbc225784520" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:21.720233 21857 raft_consensus.cc:2454] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:21.720264 22250 raft_consensus.cc:2454] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:21.720855 21924 leader_election.cc:302] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:21.721395 27393 raft_consensus.cc:2781] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:21.721541 27393 raft_consensus.cc:482] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:21.721652 27393 raft_consensus.cc:3037] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:21.722808 24866 consensus_queue.cc:237] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:21.726586 22119 raft_consensus.cc:1261] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:21.726596 24732 raft_consensus.cc:1261] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:21.727195 27515 consensus_queue.cc:1035] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:21.727980 27515 consensus_queue.cc:1035] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:21.729758 27393 raft_consensus.cc:504] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:21.730603 27395 consensus_queue.cc:1035] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:21.730952 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cda7268286458c8eaccbc225784520" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:21.731184 21857 raft_consensus.cc:3037] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:21.733615 27520 raft_consensus.cc:2932] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:21.735399 21857 raft_consensus.cc:2454] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:21.735846 21922 leader_election.cc:302] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:21.736255 27519 raft_consensus.cc:2781] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:21.736397 27519 raft_consensus.cc:686] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:54:21.736729 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cda7268286458c8eaccbc225784520" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:21.736745 27519 consensus_queue.cc:237] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:21.736963 22250 raft_consensus.cc:3037] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:21.737174 22119 raft_consensus.cc:2932] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:21.741256 22250 raft_consensus.cc:2454] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:21.744499 24732 raft_consensus.cc:2932] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:21.745359 21714 catalog_manager.cc:5484] T b60b948dae4841dd9c0919f2fcac5299 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:21.750098 24582 tablet_service.cc:1514] Processing DeleteTablet for tablet b60b948dae4841dd9c0919f2fcac5299 with delete_type TABLET_DATA_TOMBSTONED (TS 0c64d40b5d7644bd8b29feaa80eab72f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53832
I20240502 01:54:21.750823 27510 tablet_replica.cc:330] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:54:21.751019 27510 raft_consensus.cc:2229] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:21.751178 27510 raft_consensus.cc:2258] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:21.751757 27510 ts_tablet_manager.cc:1875] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:21.752857 27393 leader_election.cc:288] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:21.753497 21714 catalog_manager.cc:5484] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:21.759979 27510 ts_tablet_manager.cc:1888] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:54:21.761215 27510 log.cc:1196] T b60b948dae4841dd9c0919f2fcac5299 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/b60b948dae4841dd9c0919f2fcac5299
I20240502 01:54:21.762450 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet b60b948dae4841dd9c0919f2fcac5299 (table rebalancer_extra_table_6 [id=16e87707ee294e7c9eed52900510d340]) successfully deleted
I20240502 01:54:21.802774 21714 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:21.814473 24979 tablet_service.cc:1467] Processing CreateTablet for tablet 02a8e00ad2ad4bfa8bc1a21e50f821e4 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ea4e3391af844446873822731a91ba3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:21.814460 21967 tablet_service.cc:1467] Processing CreateTablet for tablet 02a8e00ad2ad4bfa8bc1a21e50f821e4 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ea4e3391af844446873822731a91ba3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:21.814677 24981 tablet_service.cc:1467] Processing CreateTablet for tablet d210f233014e4e80bd225256b27132a4 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ea4e3391af844446873822731a91ba3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:21.815050 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a8e00ad2ad4bfa8bc1a21e50f821e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:21.815104 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a8e00ad2ad4bfa8bc1a21e50f821e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:21.817288 24846 tablet_service.cc:1467] Processing CreateTablet for tablet d210f233014e4e80bd225256b27132a4 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ea4e3391af844446873822731a91ba3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:21.817302 24848 tablet_service.cc:1467] Processing CreateTablet for tablet 97c9eb341ce1470db791324a882a0271 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ea4e3391af844446873822731a91ba3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:21.817747 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d210f233014e4e80bd225256b27132a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:21.817962 24582 tablet_service.cc:1467] Processing CreateTablet for tablet d210f233014e4e80bd225256b27132a4 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ea4e3391af844446873822731a91ba3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:21.818087 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 97c9eb341ce1470db791324a882a0271 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ea4e3391af844446873822731a91ba3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:21.818440 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d210f233014e4e80bd225256b27132a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:21.818552 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c9eb341ce1470db791324a882a0271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:21.821466 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 97c9eb341ce1470db791324a882a0271 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ea4e3391af844446873822731a91ba3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:21.821907 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c9eb341ce1470db791324a882a0271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:21.822160 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c9eb341ce1470db791324a882a0271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:21.825280 24713 tablet_service.cc:1467] Processing CreateTablet for tablet 02a8e00ad2ad4bfa8bc1a21e50f821e4 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ea4e3391af844446873822731a91ba3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:21.825696 24713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a8e00ad2ad4bfa8bc1a21e50f821e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:21.831207 27567 tablet_bootstrap.cc:492] T d210f233014e4e80bd225256b27132a4 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:21.833118 27521 tablet_bootstrap.cc:492] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:21.833360 27567 tablet_bootstrap.cc:654] T d210f233014e4e80bd225256b27132a4 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:21.834962 27521 tablet_bootstrap.cc:654] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:21.837639 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d210f233014e4e80bd225256b27132a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:21.864746 27573 tablet_bootstrap.cc:492] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:21.865523 21714 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:21.875779 21854 raft_consensus.cc:1261] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:21.876094 27573 tablet_bootstrap.cc:654] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:21.879201 27393 consensus_queue.cc:1035] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:21.886771 22250 raft_consensus.cc:1261] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:21.892657 27393 consensus_queue.cc:1035] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:21.895294 27574 tablet_bootstrap.cc:492] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:21.897994 27574 tablet_bootstrap.cc:654] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:21.898306 27573 tablet_bootstrap.cc:492] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:21.899091 27573 ts_tablet_manager.cc:1367] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20240502 01:54:21.899597 27572 tablet_bootstrap.cc:492] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:21.900202 27573 raft_consensus.cc:348] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:21.901450 27573 raft_consensus.cc:374] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:21.901746 27573 raft_consensus.cc:729] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:21.901917 27572 tablet_bootstrap.cc:654] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:21.902348 27573 consensus_queue.cc:260] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:21.907953 27573 ts_tablet_manager.cc:1398] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:54:21.908413 27521 tablet_bootstrap.cc:492] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:21.908659 27521 ts_tablet_manager.cc:1367] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.076s user 0.006s sys 0.006s
I20240502 01:54:21.909497 27521 raft_consensus.cc:348] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:21.909755 27521 raft_consensus.cc:374] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:21.909823 27521 raft_consensus.cc:729] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:21.910101 27521 consensus_queue.cc:260] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:21.914923 27573 tablet_bootstrap.cc:492] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:21.916386 27521 ts_tablet_manager.cc:1398] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:54:21.918325 27573 tablet_bootstrap.cc:654] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:21.930738 27574 tablet_bootstrap.cc:492] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:21.931000 27574 ts_tablet_manager.cc:1367] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.004s
I20240502 01:54:21.932018 27574 raft_consensus.cc:348] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:21.932307 27574 raft_consensus.cc:374] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:21.932436 27574 raft_consensus.cc:729] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:21.932742 21712 catalog_manager.cc:5576] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093) does not have the latest schema for tablet a1cda7268286458c8eaccbc225784520 (table rebalancer_dynamic_table [id=f84e1086c99a45d396af709a16ace85f]). Expected version 1 got 0
I20240502 01:54:21.932788 27574 consensus_queue.cc:260] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:21.933547 27574 ts_tablet_manager.cc:1398] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:21.945425 27582 tablet.cc:1722] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:21.946144 27582 tablet.cc:1620] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412: MemRowSet was empty: no flush needed.
I20240502 01:54:21.949038 27572 tablet_bootstrap.cc:492] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:21.949038 27573 tablet_bootstrap.cc:492] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:21.949219 27573 ts_tablet_manager.cc:1367] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.000s
I20240502 01:54:21.949347 27572 ts_tablet_manager.cc:1367] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
I20240502 01:54:21.950073 27573 raft_consensus.cc:348] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:21.950326 27573 raft_consensus.cc:374] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:21.950407 27573 raft_consensus.cc:729] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:21.950239 27572 raft_consensus.cc:348] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:21.950664 27572 raft_consensus.cc:374] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:21.950831 27572 raft_consensus.cc:729] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:21.950696 27573 consensus_queue.cc:260] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:21.951138 27572 consensus_queue.cc:260] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:21.952682 27573 ts_tablet_manager.cc:1398] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:21.954895 27572 ts_tablet_manager.cc:1398] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:21.955219 27572 tablet_bootstrap.cc:492] T 97c9eb341ce1470db791324a882a0271 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:21.955408 27574 tablet_bootstrap.cc:492] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:21.957433 27574 tablet_bootstrap.cc:654] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:21.959208 27572 tablet_bootstrap.cc:654] T 97c9eb341ce1470db791324a882a0271 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:21.960580 27567 tablet_bootstrap.cc:492] T d210f233014e4e80bd225256b27132a4 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:21.960798 27567 ts_tablet_manager.cc:1367] T d210f233014e4e80bd225256b27132a4 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.130s user 0.017s sys 0.048s
I20240502 01:54:21.961689 27567 raft_consensus.cc:348] T d210f233014e4e80bd225256b27132a4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:21.962038 27567 raft_consensus.cc:374] T d210f233014e4e80bd225256b27132a4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:21.962173 27567 raft_consensus.cc:729] T d210f233014e4e80bd225256b27132a4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:21.962491 27567 consensus_queue.cc:260] T d210f233014e4e80bd225256b27132a4 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:21.963311 27567 ts_tablet_manager.cc:1398] T d210f233014e4e80bd225256b27132a4 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:54:21.967422 27567 tablet_bootstrap.cc:492] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:21.969036 27223 raft_consensus.cc:482] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:21.969424 27223 raft_consensus.cc:504] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:21.970090 27223 leader_election.cc:288] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:21.970407 27567 tablet_bootstrap.cc:654] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:21.970868 24866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d210f233014e4e80bd225256b27132a4" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:21.970963 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d210f233014e4e80bd225256b27132a4" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:21.971115 24866 raft_consensus.cc:2454] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:21.971168 24603 raft_consensus.cc:2454] T d210f233014e4e80bd225256b27132a4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:21.971681 24934 leader_election.cc:302] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:21.973042 27216 raft_consensus.cc:2781] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:21.973162 27216 raft_consensus.cc:482] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:21.973242 27216 raft_consensus.cc:3037] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:21.976047 27216 raft_consensus.cc:504] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:21.976589 27216 leader_election.cc:288] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:21.977610 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d210f233014e4e80bd225256b27132a4" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:21.977788 24603 raft_consensus.cc:3037] T d210f233014e4e80bd225256b27132a4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:21.980669 24603 raft_consensus.cc:2454] T d210f233014e4e80bd225256b27132a4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:21.981184 24866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d210f233014e4e80bd225256b27132a4" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:21.981298 24934 leader_election.cc:302] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:21.981416 24866 raft_consensus.cc:3037] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:21.981626 27216 raft_consensus.cc:2781] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:21.986202 24866 raft_consensus.cc:2454] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:21.987742 27583 tablet.cc:1722] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:21.991410 21712 catalog_manager.cc:5576] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093) does not have the latest schema for tablet a1cda7268286458c8eaccbc225784520 (table rebalancer_dynamic_table [id=f84e1086c99a45d396af709a16ace85f]). Expected version 1 got 0
I20240502 01:54:21.993902 27583 tablet.cc:1620] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff: MemRowSet was empty: no flush needed.
I20240502 01:54:21.998122 27572 tablet_bootstrap.cc:492] T 97c9eb341ce1470db791324a882a0271 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:21.999173 27567 tablet_bootstrap.cc:492] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:21.999395 27567 ts_tablet_manager.cc:1367] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.004s
I20240502 01:54:22.000339 27567 raft_consensus.cc:348] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:22.000682 27567 raft_consensus.cc:374] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.000805 27567 raft_consensus.cc:729] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.001075 27519 raft_consensus.cc:482] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:22.001261 27519 raft_consensus.cc:504] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:22.001999 27519 leader_election.cc:288] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:22.001298 27567 consensus_queue.cc:260] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:22.002808 27567 ts_tablet_manager.cc:1398] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:22.002707 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a8e00ad2ad4bfa8bc1a21e50f821e4" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:22.002990 25000 raft_consensus.cc:2454] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:22.003918 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a8e00ad2ad4bfa8bc1a21e50f821e4" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:22.004161 24732 raft_consensus.cc:2454] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:22.004770 21925 leader_election.cc:302] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:22.005679 27519 raft_consensus.cc:2781] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:22.005776 27519 raft_consensus.cc:482] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:22.006016 27519 raft_consensus.cc:3037] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:22.008705 21707 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=f84e1086c99a45d396af709a16ace85f] alter complete (version 1)
I20240502 01:54:22.009443 27519 raft_consensus.cc:504] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:22.011121 21714 catalog_manager.cc:5576] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785) does not have the latest schema for tablet a1cda7268286458c8eaccbc225784520 (table rebalancer_dynamic_table [id=f84e1086c99a45d396af709a16ace85f]). Expected version 1 got 0
I20240502 01:54:22.013800 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a8e00ad2ad4bfa8bc1a21e50f821e4" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:22.014082 24732 raft_consensus.cc:3037] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:22.018941 24732 raft_consensus.cc:2454] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:22.019595 21923 leader_election.cc:302] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:54:22.020191 26786 raft_consensus.cc:2781] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:22.020345 26786 raft_consensus.cc:686] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:54:22.020644 26786 consensus_queue.cc:237] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:22.025261 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a8e00ad2ad4bfa8bc1a21e50f821e4" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:22.025604 25000 raft_consensus.cc:3037] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:22.032447 25000 raft_consensus.cc:2454] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:22.033083 27519 leader_election.cc:288] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:22.035413 27572 ts_tablet_manager.cc:1367] T 97c9eb341ce1470db791324a882a0271 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.080s user 0.005s sys 0.000s
I20240502 01:54:22.040307 27595 tablet.cc:1722] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:22.042943 21712 catalog_manager.cc:5484] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:22.045146 27572 raft_consensus.cc:348] T 97c9eb341ce1470db791324a882a0271 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:22.046389 27572 raft_consensus.cc:374] T 97c9eb341ce1470db791324a882a0271 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.048527 27572 raft_consensus.cc:729] T 97c9eb341ce1470db791324a882a0271 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.049127 27572 consensus_queue.cc:260] T 97c9eb341ce1470db791324a882a0271 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:22.049980 21707 catalog_manager.cc:5576] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785) does not have the latest schema for tablet a1cda7268286458c8eaccbc225784520 (table rebalancer_dynamic_table [id=f84e1086c99a45d396af709a16ace85f]). Expected version 1 got 0
I20240502 01:54:22.052287 27595 tablet.cc:1620] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77: MemRowSet was empty: no flush needed.
I20240502 01:54:22.054039 27574 tablet_bootstrap.cc:492] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:22.054289 27574 ts_tablet_manager.cc:1367] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.121s user 0.001s sys 0.003s
I20240502 01:54:22.055397 27574 raft_consensus.cc:348] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:22.055800 27574 raft_consensus.cc:374] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.055954 27574 raft_consensus.cc:729] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.056990 27574 consensus_queue.cc:260] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:22.058590 27574 ts_tablet_manager.cc:1398] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:54:22.059082 27572 ts_tablet_manager.cc:1398] T 97c9eb341ce1470db791324a882a0271 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.024s user 0.003s sys 0.000s
I20240502 01:54:22.072196 27216 raft_consensus.cc:686] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:54:22.072707 27216 consensus_queue.cc:237] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:22.083750 21714 catalog_manager.cc:5484] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:22.117950 21710 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:22.125694 27582 tablet.cc:1722] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:22.126284 27582 tablet.cc:1620] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412: MemRowSet was empty: no flush needed.
I20240502 01:54:22.126775 27595 tablet.cc:1722] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:22.127247 27595 tablet.cc:1620] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77: MemRowSet was empty: no flush needed.
I20240502 01:54:22.130188 27583 tablet.cc:1722] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:22.130652 27583 tablet.cc:1620] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff: MemRowSet was empty: no flush needed.
I20240502 01:54:22.143129 21710 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=f84e1086c99a45d396af709a16ace85f] alter complete (version 2)
I20240502 01:54:22.209077 27260 raft_consensus.cc:482] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:22.209271 27260 raft_consensus.cc:504] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:22.209995 27260 leader_election.cc:288] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:22.210638 24866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c9eb341ce1470db791324a882a0271" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:22.210713 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c9eb341ce1470db791324a882a0271" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:22.210901 24866 raft_consensus.cc:2454] T 97c9eb341ce1470db791324a882a0271 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:22.210973 24603 raft_consensus.cc:2454] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:22.211390 24669 leader_election.cc:302] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:22.211694 27260 raft_consensus.cc:2781] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:22.211802 27260 raft_consensus.cc:482] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:22.211912 27260 raft_consensus.cc:3037] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:22.214555 27260 raft_consensus.cc:504] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:22.215538 27260 leader_election.cc:288] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:22.216166 24866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c9eb341ce1470db791324a882a0271" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:22.216226 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c9eb341ce1470db791324a882a0271" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:22.216413 24866 raft_consensus.cc:3037] T 97c9eb341ce1470db791324a882a0271 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:22.216451 24603 raft_consensus.cc:3037] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:22.219167 24866 raft_consensus.cc:2454] T 97c9eb341ce1470db791324a882a0271 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:22.219553 24669 leader_election.cc:302] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:22.220345 27389 raft_consensus.cc:2781] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:22.220484 27389 raft_consensus.cc:686] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:22.220782 27389 consensus_queue.cc:237] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:22.224964 24603 raft_consensus.cc:2454] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:22.229230 21714 catalog_manager.cc:5484] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:22.253613 21714 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:22.294984 27582 tablet.cc:1722] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:22.306313 21714 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=f84e1086c99a45d396af709a16ace85f] alter complete (version 3)
I20240502 01:54:22.307896 27595 tablet.cc:1722] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:22.322485 21836 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:22.381938 24583 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:22.406059 24979 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:22.417639 27519 consensus_queue.cc:1035] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:22.432629 27518 consensus_queue.cc:1035] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:22.434000 27583 tablet.cc:1722] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:22.448505 21714 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:22.448889 21714 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:22.478065 21967 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:54:22.503823 24713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:22.505209 21714 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet a1cda7268286458c8eaccbc225784520
I20240502 01:54:22.505466 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet a1cda7268286458c8eaccbc225784520 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:22 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:22.506224 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet a1cda7268286458c8eaccbc225784520 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:22 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:22.507308 21967 tablet_service.cc:1514] Processing DeleteTablet for tablet a1cda7268286458c8eaccbc225784520 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:22 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:22.508543 22229 tablet_service.cc:1514] Processing DeleteTablet for tablet a1cda7268286458c8eaccbc225784520 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:22 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:22.513959 22229 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:54:22.545673 27644 tablet_replica.cc:330] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:22.547560 27643 tablet_replica.cc:330] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:22.548213 27644 raft_consensus.cc:2229] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:22.549336 27644 raft_consensus.cc:2258] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:22.549839 27643 raft_consensus.cc:2229] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:22.551391 27643 raft_consensus.cc:2258] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:22.552976 27644 ts_tablet_manager.cc:1875] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:22.554432 27643 ts_tablet_manager.cc:1875] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:22.570570 27645 tablet_replica.cc:330] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:22.574808 27645 raft_consensus.cc:2229] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:22.576474 27645 raft_consensus.cc:2258] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:22.580323 27645 ts_tablet_manager.cc:1875] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:22.580370 27643 ts_tablet_manager.cc:1888] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:22.581099 27644 ts_tablet_manager.cc:1888] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:22.581336 27643 log.cc:1196] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/a1cda7268286458c8eaccbc225784520
I20240502 01:54:22.581350 27216 consensus_queue.cc:1035] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:22.582556 27644 log.cc:1196] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/a1cda7268286458c8eaccbc225784520
I20240502 01:54:22.583226 27644 ts_tablet_manager.cc:1909] T a1cda7268286458c8eaccbc225784520 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:22.583403 27643 ts_tablet_manager.cc:1909] T a1cda7268286458c8eaccbc225784520 P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
W20240502 01:54:22.586452 21701 catalog_manager.cc:4797] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): delete failed for tablet a1cda7268286458c8eaccbc225784520 because the tablet was not found. No further retry: Not found: Tablet not found: a1cda7268286458c8eaccbc225784520
I20240502 01:54:22.589768 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet a1cda7268286458c8eaccbc225784520 (table rebalancer_dynamic_table_new_name [id=f84e1086c99a45d396af709a16ace85f]) successfully deleted
I20240502 01:54:22.609917 21712 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:22.615800 22100 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:22.618609 27645 ts_tablet_manager.cc:1888] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:22.618759 27645 log.cc:1196] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/a1cda7268286458c8eaccbc225784520
I20240502 01:54:22.619168 27645 ts_tablet_manager.cc:1909] T a1cda7268286458c8eaccbc225784520 P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:22.622304 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet a1cda7268286458c8eaccbc225784520 (table rebalancer_dynamic_table_new_name [id=f84e1086c99a45d396af709a16ace85f]) successfully deleted
I20240502 01:54:22.630044 24979 tablet_service.cc:1467] Processing CreateTablet for tablet e5217d28a76e4c298a6c80741f093a60 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e2f38c9bd19c421dbcd3bc3fff2e5aa0]), partition=
I20240502 01:54:22.630652 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5217d28a76e4c298a6c80741f093a60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.630630 22231 tablet_service.cc:1467] Processing CreateTablet for tablet e5217d28a76e4c298a6c80741f093a60 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e2f38c9bd19c421dbcd3bc3fff2e5aa0]), partition=
I20240502 01:54:22.631006 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5217d28a76e4c298a6c80741f093a60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.633261 24846 tablet_service.cc:1467] Processing CreateTablet for tablet e5217d28a76e4c298a6c80741f093a60 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e2f38c9bd19c421dbcd3bc3fff2e5aa0]), partition=
I20240502 01:54:22.633675 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5217d28a76e4c298a6c80741f093a60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.661368 27216 consensus_queue.cc:1035] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:22.666232 27654 tablet_bootstrap.cc:492] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:22.672423 27652 tablet_bootstrap.cc:492] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:22.678488 27652 tablet_bootstrap.cc:654] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:22.680235 27653 tablet_bootstrap.cc:492] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:22.682122 27653 tablet_bootstrap.cc:654] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:22.675537 27654 tablet_bootstrap.cc:654] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
W20240502 01:54:22.685721 24802 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.198:42375, user_credentials={real_user=slave}} blocked reactor thread for 73087.9us
I20240502 01:54:22.697861 27653 tablet_bootstrap.cc:492] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:22.699244 27652 tablet_bootstrap.cc:492] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:22.699524 27652 ts_tablet_manager.cc:1367] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.002s
I20240502 01:54:22.700255 27652 raft_consensus.cc:348] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:22.700541 27652 raft_consensus.cc:374] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.700640 27652 raft_consensus.cc:729] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.700942 27652 consensus_queue.cc:260] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:22.702006 27653 ts_tablet_manager.cc:1367] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.001s
I20240502 01:54:22.702414 24846 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:22.702801 27652 ts_tablet_manager.cc:1398] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:22.703039 27653 raft_consensus.cc:348] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:22.703353 27653 raft_consensus.cc:374] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.703478 27653 raft_consensus.cc:729] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.703871 27653 consensus_queue.cc:260] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:22.705932 27653 ts_tablet_manager.cc:1398] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:54:22.709010 27654 tablet_bootstrap.cc:492] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:22.709275 27654 ts_tablet_manager.cc:1367] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.000s
I20240502 01:54:22.711457 27654 raft_consensus.cc:348] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:22.711836 27654 raft_consensus.cc:374] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.712247 27654 raft_consensus.cc:729] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.712626 27654 consensus_queue.cc:260] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:22.713433 27654 ts_tablet_manager.cc:1398] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:22.713658 27619 consensus_queue.cc:1035] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:22.725629 27260 consensus_queue.cc:1035] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:22.745847 27649 raft_consensus.cc:482] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:22.746053 27649 raft_consensus.cc:504] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:22.746870 27649 leader_election.cc:288] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:22.747308 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5217d28a76e4c298a6c80741f093a60" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:22.747550 24999 raft_consensus.cc:2454] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:54:22.747987 22187 leader_election.cc:302] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:22.747969 24866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5217d28a76e4c298a6c80741f093a60" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:22.748227 24866 raft_consensus.cc:2454] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:54:22.748492 27649 raft_consensus.cc:2781] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:22.748605 27649 raft_consensus.cc:482] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:22.748677 27649 raft_consensus.cc:3037] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:22.750808 27649 raft_consensus.cc:504] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:22.751416 27649 leader_election.cc:288] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:22.752672 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5217d28a76e4c298a6c80741f093a60" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:22.752655 24866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5217d28a76e4c298a6c80741f093a60" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:22.752926 24866 raft_consensus.cc:3037] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:22.752871 24999 raft_consensus.cc:3037] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:22.757714 24866 raft_consensus.cc:2454] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:54:22.758190 22185 leader_election.cc:302] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:22.758615 27649 raft_consensus.cc:2781] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:22.758787 27649 raft_consensus.cc:686] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:54:22.759300 27649 consensus_queue.cc:237] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:22.757982 24999 raft_consensus.cc:2454] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:54:22.766875 21712 catalog_manager.cc:5484] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:22.787485 24734 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "03fb607ef0d9469daca1d6f3c5259fb2"
dest_uuid: "28157bc3e678428fb5f93c077d528513"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36478
I20240502 01:54:22.787757 24734 raft_consensus.cc:595] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:22.803313 21712 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:22.816398 21967 tablet_service.cc:1467] Processing CreateTablet for tablet 4a3f86de558243eea0664d120c62f182 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b6620e4ad5274690b79a7b8e7e0523bb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:22.816951 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a3f86de558243eea0664d120c62f182. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.817497 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 4a3f86de558243eea0664d120c62f182 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b6620e4ad5274690b79a7b8e7e0523bb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:22.817562 22229 tablet_service.cc:1467] Processing CreateTablet for tablet 469abc1f70fc456b998461bd367dfc84 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b6620e4ad5274690b79a7b8e7e0523bb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:22.817963 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a3f86de558243eea0664d120c62f182. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.818150 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 469abc1f70fc456b998461bd367dfc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.821980 24979 tablet_service.cc:1467] Processing CreateTablet for tablet 469abc1f70fc456b998461bd367dfc84 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b6620e4ad5274690b79a7b8e7e0523bb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:22.822443 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 469abc1f70fc456b998461bd367dfc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.823415 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 469abc1f70fc456b998461bd367dfc84 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b6620e4ad5274690b79a7b8e7e0523bb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:22.823853 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 469abc1f70fc456b998461bd367dfc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.824498 21969 tablet_service.cc:1467] Processing CreateTablet for tablet f018db81d6214bd0ac12feaf54541fed (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b6620e4ad5274690b79a7b8e7e0523bb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:22.824875 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f018db81d6214bd0ac12feaf54541fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.825245 24981 tablet_service.cc:1467] Processing CreateTablet for tablet f018db81d6214bd0ac12feaf54541fed (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b6620e4ad5274690b79a7b8e7e0523bb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:22.825619 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f018db81d6214bd0ac12feaf54541fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.828181 21837 tablet_service.cc:1467] Processing CreateTablet for tablet f018db81d6214bd0ac12feaf54541fed (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b6620e4ad5274690b79a7b8e7e0523bb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:22.828563 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f018db81d6214bd0ac12feaf54541fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.829646 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 4a3f86de558243eea0664d120c62f182 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b6620e4ad5274690b79a7b8e7e0523bb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:22.830072 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a3f86de558243eea0664d120c62f182. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.832507 27652 tablet_bootstrap.cc:492] T 4a3f86de558243eea0664d120c62f182 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:22.833911 27652 tablet_bootstrap.cc:654] T 4a3f86de558243eea0664d120c62f182 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:22.837126 27389 raft_consensus.cc:979] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513: : Instructing follower 06543b05a16f42099bc86db2a0284412 to start an election
I20240502 01:54:22.837316 27619 raft_consensus.cc:1067] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Signalling peer 06543b05a16f42099bc86db2a0284412 to start an election
I20240502 01:54:22.838336 21986 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "03fb607ef0d9469daca1d6f3c5259fb2"
dest_uuid: "06543b05a16f42099bc86db2a0284412"
from {username='slave'} at 127.14.212.198:53247
I20240502 01:54:22.838563 21986 raft_consensus.cc:482] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:22.838657 21986 raft_consensus.cc:3037] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.842638 21986 raft_consensus.cc:504] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:22.845000 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fb607ef0d9469daca1d6f3c5259fb2" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:22.845239 21854 raft_consensus.cc:3037] T 03fb607ef0d9469daca1d6f3c5259fb2 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.847879 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fb607ef0d9469daca1d6f3c5259fb2" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:22.848126 24602 raft_consensus.cc:3037] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.848182 21854 raft_consensus.cc:2454] T 03fb607ef0d9469daca1d6f3c5259fb2 P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 2.
I20240502 01:54:22.849288 21986 leader_election.cc:288] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:22.850934 24602 raft_consensus.cc:2454] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 2.
I20240502 01:54:22.851409 21923 leader_election.cc:302] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 0c64d40b5d7644bd8b29feaa80eab72f, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:22.851742 26786 raft_consensus.cc:2781] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.851887 26786 raft_consensus.cc:686] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [term 2 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:54:22.852201 24734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fb607ef0d9469daca1d6f3c5259fb2" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:22.852154 26786 consensus_queue.cc:237] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:22.852416 27654 tablet_bootstrap.cc:492] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:22.852437 24734 raft_consensus.cc:3032] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:22.852566 24734 raft_consensus.cc:729] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:22.852864 24734 consensus_queue.cc:260] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:22.853466 24734 raft_consensus.cc:3037] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.854241 27654 tablet_bootstrap.cc:654] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:22.856676 24734 raft_consensus.cc:2454] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 2.
I20240502 01:54:22.867571 27652 tablet_bootstrap.cc:492] T 4a3f86de558243eea0664d120c62f182 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:22.867794 27652 ts_tablet_manager.cc:1367] T 4a3f86de558243eea0664d120c62f182 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20240502 01:54:22.868757 27652 raft_consensus.cc:348] T 4a3f86de558243eea0664d120c62f182 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:22.869035 27652 raft_consensus.cc:374] T 4a3f86de558243eea0664d120c62f182 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.869092 27652 raft_consensus.cc:729] T 4a3f86de558243eea0664d120c62f182 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.869295 27652 consensus_queue.cc:260] T 4a3f86de558243eea0664d120c62f182 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:22.870054 27652 ts_tablet_manager.cc:1398] T 4a3f86de558243eea0664d120c62f182 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:54:22.873592 27652 tablet_bootstrap.cc:492] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:22.875674 27680 tablet_bootstrap.cc:492] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:22.877429 21713 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:22.880481 27654 tablet_bootstrap.cc:492] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:22.880791 27654 ts_tablet_manager.cc:1367] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.000s
I20240502 01:54:22.881711 27654 raft_consensus.cc:348] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:22.882194 27654 raft_consensus.cc:374] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.882369 27654 raft_consensus.cc:729] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.884200 27654 consensus_queue.cc:260] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:22.885262 27654 ts_tablet_manager.cc:1398] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:22.885650 27654 tablet_bootstrap.cc:492] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:22.887709 27654 tablet_bootstrap.cc:654] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:22.893333 24999 raft_consensus.cc:1261] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:22.894208 27649 consensus_queue.cc:1035] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:22.895027 27683 tablet_bootstrap.cc:492] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:22.895334 27680 tablet_bootstrap.cc:654] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:22.895661 24866 raft_consensus.cc:1261] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:22.896786 27649 consensus_queue.cc:1035] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:22.900326 27683 tablet_bootstrap.cc:654] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:22.906816 27654 tablet_bootstrap.cc:492] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:22.907058 27654 ts_tablet_manager.cc:1367] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240502 01:54:22.908252 27654 raft_consensus.cc:348] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:22.908638 27654 raft_consensus.cc:374] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.908751 27654 raft_consensus.cc:729] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.909288 27678 tablet_bootstrap.cc:492] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:22.909547 27654 consensus_queue.cc:260] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:22.910893 27654 ts_tablet_manager.cc:1398] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:22.911900 27680 tablet_bootstrap.cc:492] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:22.912120 27680 ts_tablet_manager.cc:1367] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20240502 01:54:22.912966 27680 raft_consensus.cc:348] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:22.913249 27680 raft_consensus.cc:374] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.913358 27680 raft_consensus.cc:729] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.913640 27680 consensus_queue.cc:260] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:22.914251 27680 ts_tablet_manager.cc:1398] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:22.915927 27683 tablet_bootstrap.cc:492] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:22.916164 27683 ts_tablet_manager.cc:1367] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20240502 01:54:22.916993 27683 raft_consensus.cc:348] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:22.917301 27683 raft_consensus.cc:374] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.917411 27683 raft_consensus.cc:729] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.917737 27683 consensus_queue.cc:260] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:22.918123 27678 tablet_bootstrap.cc:654] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:22.919776 27683 ts_tablet_manager.cc:1398] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:22.920150 27683 tablet_bootstrap.cc:492] T 469abc1f70fc456b998461bd367dfc84 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:22.926899 27683 tablet_bootstrap.cc:654] T 469abc1f70fc456b998461bd367dfc84 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:22.937084 27652 tablet_bootstrap.cc:654] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:22.939406 21715 catalog_manager.cc:5484] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 1 to 2, leader changed from 28157bc3e678428fb5f93c077d528513 (127.14.212.198) to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 2 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:22.940461 27683 tablet_bootstrap.cc:492] T 469abc1f70fc456b998461bd367dfc84 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:22.940622 27683 ts_tablet_manager.cc:1367] T 469abc1f70fc456b998461bd367dfc84 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20240502 01:54:22.941637 27683 raft_consensus.cc:348] T 469abc1f70fc456b998461bd367dfc84 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:22.941839 27683 raft_consensus.cc:374] T 469abc1f70fc456b998461bd367dfc84 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.941910 27683 raft_consensus.cc:729] T 469abc1f70fc456b998461bd367dfc84 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.949908 27683 consensus_queue.cc:260] T 469abc1f70fc456b998461bd367dfc84 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:22.951661 27683 ts_tablet_manager.cc:1398] T 469abc1f70fc456b998461bd367dfc84 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:54:22.957056 27652 tablet_bootstrap.cc:492] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:22.957299 27652 ts_tablet_manager.cc:1367] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.001s
I20240502 01:54:22.958161 27652 raft_consensus.cc:348] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:22.958489 27652 raft_consensus.cc:374] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.958609 27652 raft_consensus.cc:729] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.958942 27652 consensus_queue.cc:260] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:22.963565 27678 tablet_bootstrap.cc:492] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:22.963778 27678 ts_tablet_manager.cc:1367] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.000s
I20240502 01:54:22.965173 27652 ts_tablet_manager.cc:1398] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:54:22.966106 27216 raft_consensus.cc:482] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:22.966310 27216 raft_consensus.cc:504] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:22.966640 27678 raft_consensus.cc:348] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:22.967063 27216 leader_election.cc:288] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:22.967501 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f018db81d6214bd0ac12feaf54541fed" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:54:22.968204 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f018db81d6214bd0ac12feaf54541fed" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:22.968441 21854 raft_consensus.cc:2454] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
W20240502 01:54:22.968569 24933 leader_election.cc:341] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:22.968732 27678 raft_consensus.cc:374] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.968900 27678 raft_consensus.cc:729] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.969226 27678 consensus_queue.cc:260] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:22.969458 24933 leader_election.cc:302] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77; no voters: 06543b05a16f42099bc86db2a0284412
I20240502 01:54:22.969779 27216 raft_consensus.cc:2781] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:22.969877 27216 raft_consensus.cc:482] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:22.969976 27216 raft_consensus.cc:3037] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:22.972707 27678 ts_tablet_manager.cc:1398] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:54:22.973094 27678 tablet_bootstrap.cc:492] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:22.974593 27678 tablet_bootstrap.cc:654] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:22.976181 27216 raft_consensus.cc:504] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:22.977190 27216 leader_election.cc:288] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:22.978349 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f018db81d6214bd0ac12feaf54541fed" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
W20240502 01:54:22.979063 24933 leader_election.cc:341] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:22.979072 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f018db81d6214bd0ac12feaf54541fed" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:22.979301 21854 raft_consensus.cc:3037] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:22.987007 21714 catalog_manager.cc:5576] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737) does not have the latest schema for tablet e5217d28a76e4c298a6c80741f093a60 (table rebalancer_dynamic_table [id=e2f38c9bd19c421dbcd3bc3fff2e5aa0]). Expected version 1 got 0
I20240502 01:54:22.987705 21854 raft_consensus.cc:2454] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:22.988162 24933 leader_election.cc:302] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77; no voters: 06543b05a16f42099bc86db2a0284412
I20240502 01:54:22.988721 27216 raft_consensus.cc:2781] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:22.988956 27216 raft_consensus.cc:686] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:54:22.989243 27216 consensus_queue.cc:237] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:22.990803 21715 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet e5217d28a76e4c298a6c80741f093a60 (table rebalancer_dynamic_table [id=e2f38c9bd19c421dbcd3bc3fff2e5aa0]). Expected version 1 got 0
I20240502 01:54:22.993377 27636 mvcc.cc:205] Tried to move back new op lower bound from 7023062478397931520 to 7023062477865103360. Current Snapshot: MvccSnapshot[applied={T|T < 7023062478397931520}]
I20240502 01:54:22.994091 27696 tablet.cc:1722] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:22.995760 27696 tablet.cc:1620] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:23.007988 21715 catalog_manager.cc:5484] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:23.009239 27697 tablet.cc:1722] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:23.010228 27697 tablet.cc:1620] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:23.011891 21712 catalog_manager.cc:5576] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737) does not have the latest schema for tablet e5217d28a76e4c298a6c80741f093a60 (table rebalancer_dynamic_table [id=e2f38c9bd19c421dbcd3bc3fff2e5aa0]). Expected version 1 got 0
I20240502 01:54:23.014701 21715 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=e2f38c9bd19c421dbcd3bc3fff2e5aa0] alter complete (version 1)
I20240502 01:54:23.018741 27694 tablet.cc:1722] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:23.021217 27694 tablet.cc:1620] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff: MemRowSet was empty: no flush needed.
I20240502 01:54:23.022675 27678 tablet_bootstrap.cc:492] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:23.022822 27678 ts_tablet_manager.cc:1367] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.050s user 0.013s sys 0.010s
I20240502 01:54:23.023386 27678 raft_consensus.cc:348] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:23.023566 27678 raft_consensus.cc:374] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:23.023661 27678 raft_consensus.cc:729] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:23.023885 27678 consensus_queue.cc:260] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:23.024360 27678 ts_tablet_manager.cc:1398] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 01:54:23.032740 27207 raft_consensus.cc:482] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:23.032944 27207 raft_consensus.cc:504] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:23.033915 27207 leader_election.cc:288] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:23.034320 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a3f86de558243eea0664d120c62f182" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:54:23.034375 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a3f86de558243eea0664d120c62f182" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:23.034564 21986 raft_consensus.cc:2454] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:54:23.034694 22249 raft_consensus.cc:2454] T 4a3f86de558243eea0664d120c62f182 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:54:23.035112 22053 leader_election.cc:302] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:54:23.035378 27207 raft_consensus.cc:2781] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:23.035459 27207 raft_consensus.cc:482] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:23.035512 27207 raft_consensus.cc:3037] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.038322 27207 raft_consensus.cc:504] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:23.039218 27207 leader_election.cc:288] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:23.039521 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a3f86de558243eea0664d120c62f182" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:54:23.039536 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a3f86de558243eea0664d120c62f182" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:23.039690 21986 raft_consensus.cc:3037] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.039736 22249 raft_consensus.cc:3037] T 4a3f86de558243eea0664d120c62f182 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.042202 22249 raft_consensus.cc:2454] T 4a3f86de558243eea0664d120c62f182 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:54:23.042641 22055 leader_election.cc:302] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:54:23.042887 27207 raft_consensus.cc:2781] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:23.042991 27207 raft_consensus.cc:686] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:54:23.043219 27207 consensus_queue.cc:237] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:23.044379 21986 raft_consensus.cc:2454] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:54:23.051196 21712 catalog_manager.cc:5484] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:23.057917 26786 raft_consensus.cc:482] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:23.058063 26786 raft_consensus.cc:504] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:23.058797 26786 leader_election.cc:288] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:23.059345 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f018db81d6214bd0ac12feaf54541fed" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:23.059643 21854 raft_consensus.cc:2379] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 06543b05a16f42099bc86db2a0284412 in current term 1: Already voted for candidate 41c70f373c9d49a49891b03a12c6cb2b in this term.
I20240502 01:54:23.060370 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f018db81d6214bd0ac12feaf54541fed" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:23.062080 21925 leader_election.cc:302] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06543b05a16f42099bc86db2a0284412; no voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77
I20240502 01:54:23.063118 26786 raft_consensus.cc:3037] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.066547 26786 raft_consensus.cc:2726] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:23.118918 21710 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:23.139569 27694 tablet.cc:1722] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:23.140120 27694 tablet.cc:1620] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff: MemRowSet was empty: no flush needed.
I20240502 01:54:23.141098 27697 tablet.cc:1722] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:23.141499 27697 tablet.cc:1620] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:23.144433 27696 tablet.cc:1722] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:23.145012 27696 tablet.cc:1620] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:23.153416 27655 raft_consensus.cc:482] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:23.153594 27655 raft_consensus.cc:504] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:23.154385 27655 leader_election.cc:288] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:23.154879 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "469abc1f70fc456b998461bd367dfc84" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:23.155143 21857 raft_consensus.cc:2454] T 469abc1f70fc456b998461bd367dfc84 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:23.159281 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "469abc1f70fc456b998461bd367dfc84" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:23.159503 22249 raft_consensus.cc:2454] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:23.160010 24936 leader_election.cc:302] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:23.160840 27651 raft_consensus.cc:2781] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:23.160998 27651 raft_consensus.cc:482] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:23.161147 27651 raft_consensus.cc:3037] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.165546 21708 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=e2f38c9bd19c421dbcd3bc3fff2e5aa0] alter complete (version 2)
I20240502 01:54:23.167811 27651 raft_consensus.cc:504] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:23.168830 27651 leader_election.cc:288] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:23.169451 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "469abc1f70fc456b998461bd367dfc84" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:23.169662 22249 raft_consensus.cc:3037] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.172458 22249 raft_consensus.cc:2454] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:23.174567 24936 leader_election.cc:302] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:23.176746 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "469abc1f70fc456b998461bd367dfc84" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:23.177016 21857 raft_consensus.cc:3037] T 469abc1f70fc456b998461bd367dfc84 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.178434 27655 raft_consensus.cc:2781] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:23.178617 27655 raft_consensus.cc:686] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:54:23.178925 27655 consensus_queue.cc:237] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:23.182103 21857 raft_consensus.cc:2454] T 469abc1f70fc456b998461bd367dfc84 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:23.191475 21713 catalog_manager.cc:5484] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:23.270488 21713 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:23.278200 24583 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:23.283130 21967 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:54:23.293118 27694 tablet.cc:1722] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:23.298960 27697 tablet.cc:1722] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:23.304744 27696 tablet.cc:1722] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:23.319031 21712 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=e2f38c9bd19c421dbcd3bc3fff2e5aa0] alter complete (version 3)
I20240502 01:54:23.331223 24602 raft_consensus.cc:1261] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:54:23.331987 26786 consensus_queue.cc:1035] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 01:54:23.352217 21925 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.200:37097, user_credentials={real_user=slave}} blocked reactor thread for 91037.5us
I20240502 01:54:23.362511 24713 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:23.385246 27651 consensus_queue.cc:1035] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:23.385907 27651 consensus_queue.cc:1035] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:23.424561 21713 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:23.424901 21713 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:23.428601 21713 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet e5217d28a76e4c298a6c80741f093a60
I20240502 01:54:23.430472 22229 tablet_service.cc:1514] Processing DeleteTablet for tablet e5217d28a76e4c298a6c80741f093a60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:23 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:23.431545 24846 tablet_service.cc:1514] Processing DeleteTablet for tablet e5217d28a76e4c298a6c80741f093a60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:23 UTC) from {username='slave'} at 127.0.0.1:51592
I20240502 01:54:23.433717 21837 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:23.445361 24732 raft_consensus.cc:1261] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:54:23.445945 27738 consensus_queue.cc:1035] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:23.448606 24979 tablet_service.cc:1514] Processing DeleteTablet for tablet e5217d28a76e4c298a6c80741f093a60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:23 UTC) from {username='slave'} at 127.0.0.1:39964
I20240502 01:54:23.466125 22100 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:23.474784 21857 raft_consensus.cc:1261] T 03fb607ef0d9469daca1d6f3c5259fb2 P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:54:23.475389 26786 consensus_queue.cc:1035] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:23.495713 22231 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:23.503793 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:54:23.517616 27754 tablet_replica.cc:330] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08: stopping tablet replica
I20240502 01:54:23.517894 27754 raft_consensus.cc:2229] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:23.518163 27754 raft_consensus.cc:2258] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:23.518694 27753 tablet_replica.cc:330] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:23.518914 27753 raft_consensus.cc:2229] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:23.519281 27753 raft_consensus.cc:2258] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:23.521931 27756 tablet_replica.cc:330] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:54:23.522153 27756 raft_consensus.cc:2229] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:23.522398 27756 raft_consensus.cc:2258] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:23.531351 21706 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:23.533634 27753 ts_tablet_manager.cc:1875] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:23.536214 27754 ts_tablet_manager.cc:1875] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:23.539395 27756 ts_tablet_manager.cc:1875] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:23.553740 24979 tablet_service.cc:1467] Processing CreateTablet for tablet c27c4dfead60411d8806dd8789759d57 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b615071706664c5a8eeb7a798e00d7ac]), partition=
I20240502 01:54:23.554817 21837 tablet_service.cc:1467] Processing CreateTablet for tablet c27c4dfead60411d8806dd8789759d57 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b615071706664c5a8eeb7a798e00d7ac]), partition=
I20240502 01:54:23.555245 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c27c4dfead60411d8806dd8789759d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:23.559824 24583 tablet_service.cc:1467] Processing CreateTablet for tablet c27c4dfead60411d8806dd8789759d57 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b615071706664c5a8eeb7a798e00d7ac]), partition=
I20240502 01:54:23.560212 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c27c4dfead60411d8806dd8789759d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:23.571365 27754 ts_tablet_manager.cc:1888] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:23.573055 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c27c4dfead60411d8806dd8789759d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:23.578097 27753 ts_tablet_manager.cc:1888] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:23.579555 27754 log.cc:1196] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/wals/e5217d28a76e4c298a6c80741f093a60
I20240502 01:54:23.578327 27753 log.cc:1196] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/e5217d28a76e4c298a6c80741f093a60
I20240502 01:54:23.580118 27753 ts_tablet_manager.cc:1909] T e5217d28a76e4c298a6c80741f093a60 P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:23.580672 27754 ts_tablet_manager.cc:1909] T e5217d28a76e4c298a6c80741f093a60 P d7449bb1df214bf4a5f93e4631312f08: Deleting consensus metadata
I20240502 01:54:23.582046 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet e5217d28a76e4c298a6c80741f093a60 (table rebalancer_dynamic_table_new_name [id=e2f38c9bd19c421dbcd3bc3fff2e5aa0]) successfully deleted
I20240502 01:54:23.587203 21700 catalog_manager.cc:4830] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): tablet e5217d28a76e4c298a6c80741f093a60 (table rebalancer_dynamic_table_new_name [id=e2f38c9bd19c421dbcd3bc3fff2e5aa0]) successfully deleted
I20240502 01:54:23.591190 27756 ts_tablet_manager.cc:1888] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:23.591418 27756 log.cc:1196] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/e5217d28a76e4c298a6c80741f093a60
I20240502 01:54:23.592006 27756 ts_tablet_manager.cc:1909] T e5217d28a76e4c298a6c80741f093a60 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting consensus metadata
I20240502 01:54:23.593112 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet e5217d28a76e4c298a6c80741f093a60 (table rebalancer_dynamic_table_new_name [id=e2f38c9bd19c421dbcd3bc3fff2e5aa0]) successfully deleted
I20240502 01:54:23.586925 27764 tablet_bootstrap.cc:492] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:23.602885 27764 tablet_bootstrap.cc:654] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:23.613179 24846 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:23.629953 27766 tablet_bootstrap.cc:492] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:23.633447 27765 tablet_bootstrap.cc:492] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:23.633551 27766 tablet_bootstrap.cc:654] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:23.635371 27765 tablet_bootstrap.cc:654] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:23.638116 27764 tablet_bootstrap.cc:492] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:23.638785 27764 ts_tablet_manager.cc:1367] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
I20240502 01:54:23.639715 27764 raft_consensus.cc:348] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:23.639918 27207 consensus_queue.cc:1035] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:23.640062 27764 raft_consensus.cc:374] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:23.640167 27764 raft_consensus.cc:729] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:23.640416 27764 consensus_queue.cc:260] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:23.641227 27764 ts_tablet_manager.cc:1398] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:23.654887 27207 consensus_queue.cc:1035] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:23.659925 27765 tablet_bootstrap.cc:492] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:23.660166 27765 ts_tablet_manager.cc:1367] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240502 01:54:23.661070 27765 raft_consensus.cc:348] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:23.661397 27765 raft_consensus.cc:374] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:23.661530 27765 raft_consensus.cc:729] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:23.661834 27765 consensus_queue.cc:260] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:23.664469 27765 ts_tablet_manager.cc:1398] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:23.670450 27655 consensus_queue.cc:1035] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:23.683573 27766 tablet_bootstrap.cc:492] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:23.683876 27766 ts_tablet_manager.cc:1367] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.000s
I20240502 01:54:23.684890 27766 raft_consensus.cc:348] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:23.685138 27766 raft_consensus.cc:374] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:23.685231 27766 raft_consensus.cc:729] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:23.685505 27766 consensus_queue.cc:260] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:23.686241 27766 ts_tablet_manager.cc:1398] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:23.697065 27516 raft_consensus.cc:482] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:23.697283 27516 raft_consensus.cc:504] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:23.697425 27655 consensus_queue.cc:1035] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:23.698351 27516 leader_election.cc:288] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:23.699189 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27c4dfead60411d8806dd8789759d57" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:23.699422 24603 raft_consensus.cc:2454] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:54:23.699604 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27c4dfead60411d8806dd8789759d57" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:23.699868 24999 raft_consensus.cc:2454] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:54:23.699921 21791 leader_election.cc:302] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:23.700564 27516 raft_consensus.cc:2781] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:23.700685 27516 raft_consensus.cc:482] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:23.700788 27516 raft_consensus.cc:3037] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.703994 21706 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:23.707374 27516 raft_consensus.cc:504] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:23.708525 27516 leader_election.cc:288] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:23.709370 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27c4dfead60411d8806dd8789759d57" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:23.709600 24999 raft_consensus.cc:3037] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.712271 24999 raft_consensus.cc:2454] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:54:23.712716 21793 leader_election.cc:302] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:23.713323 27516 raft_consensus.cc:2781] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:23.713487 27516 raft_consensus.cc:686] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:54:23.713783 27516 consensus_queue.cc:237] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:23.713951 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27c4dfead60411d8806dd8789759d57" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:23.714195 24602 raft_consensus.cc:3037] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.719324 21837 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6a655aeff34e1a89d91aae14510355 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=7771fca28431450880af2d1b69228f23]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:23.719751 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6a655aeff34e1a89d91aae14510355. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:23.724503 27766 tablet_bootstrap.cc:492] T 7c6a655aeff34e1a89d91aae14510355 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:23.725940 24713 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6a655aeff34e1a89d91aae14510355 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=7771fca28431450880af2d1b69228f23]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:23.726451 24713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6a655aeff34e1a89d91aae14510355. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:23.726418 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6a655aeff34e1a89d91aae14510355 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=7771fca28431450880af2d1b69228f23]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:23.726823 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6a655aeff34e1a89d91aae14510355. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:23.727048 27766 tablet_bootstrap.cc:654] T 7c6a655aeff34e1a89d91aae14510355 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:23.729312 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 458b19e5186f4ce79f618a10e101c605 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=7771fca28431450880af2d1b69228f23]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:23.729661 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458b19e5186f4ce79f618a10e101c605. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:23.731031 24846 tablet_service.cc:1467] Processing CreateTablet for tablet 458b19e5186f4ce79f618a10e101c605 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=7771fca28431450880af2d1b69228f23]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:23.731443 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458b19e5186f4ce79f618a10e101c605. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:23.733485 24582 tablet_service.cc:1467] Processing CreateTablet for tablet 7812450aabc54e1db36799d8e030d1ed (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=7771fca28431450880af2d1b69228f23]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:23.733903 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7812450aabc54e1db36799d8e030d1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:23.733975 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 7812450aabc54e1db36799d8e030d1ed (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=7771fca28431450880af2d1b69228f23]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:23.734431 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7812450aabc54e1db36799d8e030d1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:23.737277 22229 tablet_service.cc:1467] Processing CreateTablet for tablet 458b19e5186f4ce79f618a10e101c605 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=7771fca28431450880af2d1b69228f23]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:23.737746 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458b19e5186f4ce79f618a10e101c605. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:23.738510 21967 tablet_service.cc:1467] Processing CreateTablet for tablet 7812450aabc54e1db36799d8e030d1ed (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=7771fca28431450880af2d1b69228f23]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:23.738907 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7812450aabc54e1db36799d8e030d1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:23.739106 27766 tablet_bootstrap.cc:492] T 7c6a655aeff34e1a89d91aae14510355 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:23.739262 27766 ts_tablet_manager.cc:1367] T 7c6a655aeff34e1a89d91aae14510355 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240502 01:54:23.740029 27766 raft_consensus.cc:348] T 7c6a655aeff34e1a89d91aae14510355 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:23.740227 27766 raft_consensus.cc:374] T 7c6a655aeff34e1a89d91aae14510355 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:23.740289 27766 raft_consensus.cc:729] T 7c6a655aeff34e1a89d91aae14510355 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:23.740522 27766 consensus_queue.cc:260] T 7c6a655aeff34e1a89d91aae14510355 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:23.741079 27766 ts_tablet_manager.cc:1398] T 7c6a655aeff34e1a89d91aae14510355 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:23.749022 27780 tablet_bootstrap.cc:492] T 7812450aabc54e1db36799d8e030d1ed P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:23.751458 27780 tablet_bootstrap.cc:654] T 7812450aabc54e1db36799d8e030d1ed P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:23.752756 27764 tablet_bootstrap.cc:492] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:23.754252 27780 tablet_bootstrap.cc:492] T 7812450aabc54e1db36799d8e030d1ed P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:23.754462 27764 tablet_bootstrap.cc:654] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:23.754457 27780 ts_tablet_manager.cc:1367] T 7812450aabc54e1db36799d8e030d1ed P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240502 01:54:23.755416 27209 raft_consensus.cc:482] T 7c6a655aeff34e1a89d91aae14510355 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:23.755342 27780 raft_consensus.cc:348] T 7812450aabc54e1db36799d8e030d1ed P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:23.755692 27780 raft_consensus.cc:374] T 7812450aabc54e1db36799d8e030d1ed P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:23.755578 27209 raft_consensus.cc:504] T 7c6a655aeff34e1a89d91aae14510355 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:23.755822 27780 raft_consensus.cc:729] T 7812450aabc54e1db36799d8e030d1ed P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:23.756172 27780 consensus_queue.cc:260] T 7812450aabc54e1db36799d8e030d1ed P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:23.756532 27209 leader_election.cc:288] T 7c6a655aeff34e1a89d91aae14510355 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:54:23.757369 21791 leader_election.cc:341] T 7c6a655aeff34e1a89d91aae14510355 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 7c6a655aeff34e1a89d91aae14510355
W20240502 01:54:23.757748 21791 leader_election.cc:341] T 7c6a655aeff34e1a89d91aae14510355 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Not found: Tablet not found: 7c6a655aeff34e1a89d91aae14510355
I20240502 01:54:23.757848 21791 leader_election.cc:302] T 7c6a655aeff34e1a89d91aae14510355 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513
I20240502 01:54:23.758114 27209 raft_consensus.cc:2726] T 7c6a655aeff34e1a89d91aae14510355 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:23.759891 27780 ts_tablet_manager.cc:1398] T 7812450aabc54e1db36799d8e030d1ed P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:54:23.785740 27781 tablet_bootstrap.cc:492] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:23.787592 27781 tablet_bootstrap.cc:654] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:23.799458 27518 raft_consensus.cc:482] T 7812450aabc54e1db36799d8e030d1ed P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:23.799638 27518 raft_consensus.cc:504] T 7812450aabc54e1db36799d8e030d1ed P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:23.800516 27518 leader_election.cc:288] T 7812450aabc54e1db36799d8e030d1ed P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:23.801134 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7812450aabc54e1db36799d8e030d1ed" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
W20240502 01:54:23.801728 21924 leader_election.cc:341] T 7812450aabc54e1db36799d8e030d1ed P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:23.804349 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7812450aabc54e1db36799d8e030d1ed" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
W20240502 01:54:23.804968 21923 leader_election.cc:341] T 7812450aabc54e1db36799d8e030d1ed P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:23.805102 21923 leader_election.cc:302] T 7812450aabc54e1db36799d8e030d1ed P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06543b05a16f42099bc86db2a0284412; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, c64840657cdf4b93980bd34120c94d9a
I20240502 01:54:23.805299 27789 tablet_bootstrap.cc:492] T 7c6a655aeff34e1a89d91aae14510355 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:23.805622 27761 raft_consensus.cc:2726] T 7812450aabc54e1db36799d8e030d1ed P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:23.809618 27789 tablet_bootstrap.cc:654] T 7c6a655aeff34e1a89d91aae14510355 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:23.810941 24602 raft_consensus.cc:2454] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:54:23.818405 27789 tablet_bootstrap.cc:492] T 7c6a655aeff34e1a89d91aae14510355 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:23.818804 27789 ts_tablet_manager.cc:1367] T 7c6a655aeff34e1a89d91aae14510355 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20240502 01:54:23.820168 27789 raft_consensus.cc:348] T 7c6a655aeff34e1a89d91aae14510355 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:23.821213 27789 raft_consensus.cc:374] T 7c6a655aeff34e1a89d91aae14510355 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:23.821522 27789 raft_consensus.cc:729] T 7c6a655aeff34e1a89d91aae14510355 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:23.822098 27789 consensus_queue.cc:260] T 7c6a655aeff34e1a89d91aae14510355 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:23.825237 27789 ts_tablet_manager.cc:1398] T 7c6a655aeff34e1a89d91aae14510355 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 01:54:23.825891 27789 tablet_bootstrap.cc:492] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:23.828711 27789 tablet_bootstrap.cc:654] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:23.843763 27786 tablet_bootstrap.cc:492] T 7812450aabc54e1db36799d8e030d1ed P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:23.845703 27786 tablet_bootstrap.cc:654] T 7812450aabc54e1db36799d8e030d1ed P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:23.850472 27764 tablet_bootstrap.cc:492] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:23.851711 27764 ts_tablet_manager.cc:1367] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.099s user 0.018s sys 0.010s
I20240502 01:54:23.851830 27789 tablet_bootstrap.cc:492] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:23.852139 27787 tablet_bootstrap.cc:492] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:23.852535 27789 ts_tablet_manager.cc:1367] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240502 01:54:23.853173 27764 raft_consensus.cc:348] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:23.854239 27787 tablet_bootstrap.cc:654] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:23.855527 27764 raft_consensus.cc:374] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:23.855991 27764 raft_consensus.cc:729] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:23.856791 27764 consensus_queue.cc:260] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:23.859953 27764 ts_tablet_manager.cc:1398] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:54:23.861250 27764 tablet_bootstrap.cc:492] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:23.865252 27764 tablet_bootstrap.cc:654] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:23.869233 27789 raft_consensus.cc:348] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:23.869910 27789 raft_consensus.cc:374] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:23.870086 27789 raft_consensus.cc:729] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:23.870927 27789 consensus_queue.cc:260] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:23.873575 27781 tablet_bootstrap.cc:492] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:23.873780 27781 ts_tablet_manager.cc:1367] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.088s user 0.012s sys 0.006s
I20240502 01:54:23.874554 27781 raft_consensus.cc:348] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:23.874825 27781 raft_consensus.cc:374] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:23.874915 27781 raft_consensus.cc:729] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:23.875138 27781 consensus_queue.cc:260] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:23.876333 27789 ts_tablet_manager.cc:1398] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240502 01:54:23.876789 27781 ts_tablet_manager.cc:1398] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:23.877319 27786 tablet_bootstrap.cc:492] T 7812450aabc54e1db36799d8e030d1ed P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:23.878518 27764 tablet_bootstrap.cc:492] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:23.880304 27764 ts_tablet_manager.cc:1367] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20240502 01:54:23.881286 27786 ts_tablet_manager.cc:1367] T 7812450aabc54e1db36799d8e030d1ed P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.004s
I20240502 01:54:23.881305 27764 raft_consensus.cc:348] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:23.882167 27786 raft_consensus.cc:348] T 7812450aabc54e1db36799d8e030d1ed P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:23.882480 27764 raft_consensus.cc:374] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:23.882712 27764 raft_consensus.cc:729] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:23.883168 27764 consensus_queue.cc:260] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:23.884953 27786 raft_consensus.cc:374] T 7812450aabc54e1db36799d8e030d1ed P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:23.885195 27786 raft_consensus.cc:729] T 7812450aabc54e1db36799d8e030d1ed P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:23.885529 27764 ts_tablet_manager.cc:1398] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:54:23.885560 27786 consensus_queue.cc:260] T 7812450aabc54e1db36799d8e030d1ed P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:23.887307 27786 ts_tablet_manager.cc:1398] T 7812450aabc54e1db36799d8e030d1ed P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 01:54:23.903767 27787 tablet_bootstrap.cc:492] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:23.904017 27336 raft_consensus.cc:482] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:23.904207 27336 raft_consensus.cc:504] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:23.905076 27336 leader_election.cc:288] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:23.905817 21987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7812450aabc54e1db36799d8e030d1ed" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:54:23.906028 21987 raft_consensus.cc:2454] T 7812450aabc54e1db36799d8e030d1ed P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:23.906594 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7812450aabc54e1db36799d8e030d1ed" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:54:23.906798 22119 raft_consensus.cc:2454] T 7812450aabc54e1db36799d8e030d1ed P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:23.907574 27787 ts_tablet_manager.cc:1367] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.057s user 0.006s sys 0.000s
I20240502 01:54:23.907805 24536 leader_election.cc:302] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 0c64d40b5d7644bd8b29feaa80eab72f; no voters:
I20240502 01:54:23.908416 27787 raft_consensus.cc:348] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:23.908627 27787 raft_consensus.cc:374] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:23.908701 27787 raft_consensus.cc:729] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:23.909410 27787 consensus_queue.cc:260] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:23.909962 27787 ts_tablet_manager.cc:1398] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:23.915588 27568 raft_consensus.cc:2781] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:23.915719 27568 raft_consensus.cc:482] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:23.915827 27568 raft_consensus.cc:3037] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.918993 27568 raft_consensus.cc:504] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:23.920150 21987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7812450aabc54e1db36799d8e030d1ed" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:54:23.920141 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7812450aabc54e1db36799d8e030d1ed" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:54:23.920349 21987 raft_consensus.cc:3037] T 7812450aabc54e1db36799d8e030d1ed P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.920385 22119 raft_consensus.cc:3037] T 7812450aabc54e1db36799d8e030d1ed P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.911860 21706 catalog_manager.cc:5484] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:23.923074 21987 raft_consensus.cc:2454] T 7812450aabc54e1db36799d8e030d1ed P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:23.923089 22119 raft_consensus.cc:2454] T 7812450aabc54e1db36799d8e030d1ed P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:23.923637 24538 leader_election.cc:302] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:54:23.923919 27336 raft_consensus.cc:2781] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:23.924067 27568 leader_election.cc:288] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:23.943799 27336 raft_consensus.cc:686] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:54:23.944242 27336 consensus_queue.cc:237] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:23.961001 21706 catalog_manager.cc:5484] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:24.086282 27650 raft_consensus.cc:482] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:24.086483 27650 raft_consensus.cc:504] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:24.087347 27650 leader_election.cc:288] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:24.088395 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458b19e5186f4ce79f618a10e101c605" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:24.088439 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458b19e5186f4ce79f618a10e101c605" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:24.088649 22250 raft_consensus.cc:2454] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:24.088742 24732 raft_consensus.cc:2454] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:24.089162 24803 leader_election.cc:302] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:24.090240 27650 raft_consensus.cc:2781] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:24.090351 27650 raft_consensus.cc:482] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:24.090469 27650 raft_consensus.cc:3037] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:24.092445 27650 raft_consensus.cc:504] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:24.093297 27650 leader_election.cc:288] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:24.093739 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458b19e5186f4ce79f618a10e101c605" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:24.093916 22250 raft_consensus.cc:3037] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:24.094936 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458b19e5186f4ce79f618a10e101c605" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:24.095134 24732 raft_consensus.cc:3037] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:24.096724 22250 raft_consensus.cc:2454] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:24.100689 24732 raft_consensus.cc:2454] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:24.103786 24803 leader_election.cc:302] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:24.104590 27640 raft_consensus.cc:2781] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:24.104743 27640 raft_consensus.cc:686] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:24.105067 27640 consensus_queue.cc:237] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:24.106245 21713 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:24.114964 24999 raft_consensus.cc:1261] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:24.115052 24602 raft_consensus.cc:1261] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:24.115656 27516 consensus_queue.cc:1035] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:24.116117 27785 consensus_queue.cc:1035] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:24.122479 27835 tablet.cc:1722] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:24.123414 27835 tablet.cc:1620] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77: MemRowSet was empty: no flush needed.
I20240502 01:54:24.126478 27336 raft_consensus.cc:482] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:24.126701 27336 raft_consensus.cc:504] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:24.127647 27336 leader_election.cc:288] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:24.129180 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6a655aeff34e1a89d91aae14510355" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:24.129496 24732 raft_consensus.cc:2454] T 7c6a655aeff34e1a89d91aae14510355 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:24.129484 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6a655aeff34e1a89d91aae14510355" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:24.129714 21856 raft_consensus.cc:2454] T 7c6a655aeff34e1a89d91aae14510355 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:24.130100 24537 leader_election.cc:302] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:54:24.131803 27336 raft_consensus.cc:2781] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:24.131964 27336 raft_consensus.cc:482] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:24.132073 27336 raft_consensus.cc:3037] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:24.137316 27336 raft_consensus.cc:504] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:24.139181 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6a655aeff34e1a89d91aae14510355" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:24.139444 21856 raft_consensus.cc:3037] T 7c6a655aeff34e1a89d91aae14510355 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:24.145252 21708 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet c27c4dfead60411d8806dd8789759d57 (table rebalancer_dynamic_table [id=b615071706664c5a8eeb7a798e00d7ac]). Expected version 1 got 0
I20240502 01:54:24.145959 27336 leader_election.cc:288] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:24.144840 27837 tablet.cc:1722] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:24.146435 27837 tablet.cc:1620] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:24.149546 21707 catalog_manager.cc:5484] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:24.154011 21714 catalog_manager.cc:5576] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677) does not have the latest schema for tablet c27c4dfead60411d8806dd8789759d57 (table rebalancer_dynamic_table [id=b615071706664c5a8eeb7a798e00d7ac]). Expected version 1 got 0
I20240502 01:54:24.156514 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6a655aeff34e1a89d91aae14510355" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:24.156785 24732 raft_consensus.cc:3037] T 7c6a655aeff34e1a89d91aae14510355 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:24.158548 21713 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=b615071706664c5a8eeb7a798e00d7ac] alter complete (version 1)
I20240502 01:54:24.158521 27838 tablet.cc:1722] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:24.159739 27838 tablet.cc:1620] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:24.161846 21856 raft_consensus.cc:2454] T 7c6a655aeff34e1a89d91aae14510355 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:24.163077 24536 leader_election.cc:302] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:24.163529 27795 raft_consensus.cc:2781] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:24.163693 27795 raft_consensus.cc:686] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:54:24.164001 27795 consensus_queue.cc:237] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:24.163928 24732 raft_consensus.cc:2454] T 7c6a655aeff34e1a89d91aae14510355 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:24.194003 21707 catalog_manager.cc:5484] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:24.262425 21713 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:24.283085 27835 tablet.cc:1722] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:24.283615 27835 tablet.cc:1620] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77: MemRowSet was empty: no flush needed.
I20240502 01:54:24.287748 27838 tablet.cc:1722] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:24.288264 27838 tablet.cc:1620] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:24.292142 27837 tablet.cc:1722] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:24.292667 27837 tablet.cc:1620] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:24.298192 21714 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=b615071706664c5a8eeb7a798e00d7ac] alter complete (version 2)
I20240502 01:54:24.344990 27795 consensus_queue.cc:1035] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:24.384651 24583 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:24.411393 21707 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:24.424597 27835 tablet.cc:1722] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:24.429433 27837 tablet.cc:1722] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:24.440222 27838 tablet.cc:1722] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:24.443279 21714 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=b615071706664c5a8eeb7a798e00d7ac] alter complete (version 3)
I20240502 01:54:24.447968 24713 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:24.465498 27568 consensus_queue.cc:1035] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:24.469126 21967 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:54:24.516211 22229 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:54:24.532509 21837 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:24.541752 24979 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:54:24.561524 21707 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:24.574105 27640 consensus_queue.cc:1035] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:24.575381 27568 consensus_queue.cc:1035] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:24.588565 21707 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:24.593014 24846 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:24.607192 21707 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet c27c4dfead60411d8806dd8789759d57
I20240502 01:54:24.620126 22100 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:24.649451 21837 tablet_service.cc:1514] Processing DeleteTablet for tablet c27c4dfead60411d8806dd8789759d57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:24 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:24.650168 24583 tablet_service.cc:1514] Processing DeleteTablet for tablet c27c4dfead60411d8806dd8789759d57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:24 UTC) from {username='slave'} at 127.0.0.1:53832
I20240502 01:54:24.650818 24979 tablet_service.cc:1514] Processing DeleteTablet for tablet c27c4dfead60411d8806dd8789759d57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:24 UTC) from {username='slave'} at 127.0.0.1:39964
I20240502 01:54:24.655404 27881 tablet_replica.cc:330] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:54:24.656569 27881 raft_consensus.cc:2229] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:24.660809 27880 tablet_replica.cc:330] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:54:24.661176 27881 raft_consensus.cc:2258] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:24.661654 27880 raft_consensus.cc:2229] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:24.662763 27880 raft_consensus.cc:2258] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:24.662488 27881 ts_tablet_manager.cc:1875] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:24.662969 27795 consensus_queue.cc:1035] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:24.666311 27880 ts_tablet_manager.cc:1875] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:24.672751 27879 tablet_replica.cc:330] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:24.673151 27879 raft_consensus.cc:2229] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:24.673568 27879 raft_consensus.cc:2258] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:24.674624 27879 ts_tablet_manager.cc:1875] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:24.674641 27650 consensus_queue.cc:1035] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:24.675331 27881 ts_tablet_manager.cc:1888] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:24.681001 27881 log.cc:1196] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/c27c4dfead60411d8806dd8789759d57
I20240502 01:54:24.681660 27881 ts_tablet_manager.cc:1909] T c27c4dfead60411d8806dd8789759d57 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting consensus metadata
I20240502 01:54:24.682730 27880 ts_tablet_manager.cc:1888] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:24.684787 27880 log.cc:1196] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/c27c4dfead60411d8806dd8789759d57
I20240502 01:54:24.685395 27880 ts_tablet_manager.cc:1909] T c27c4dfead60411d8806dd8789759d57 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting consensus metadata
I20240502 01:54:24.686501 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet c27c4dfead60411d8806dd8789759d57 (table rebalancer_dynamic_table_new_name [id=b615071706664c5a8eeb7a798e00d7ac]) successfully deleted
I20240502 01:54:24.689137 27879 ts_tablet_manager.cc:1888] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:24.689674 27879 log.cc:1196] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/c27c4dfead60411d8806dd8789759d57
I20240502 01:54:24.690171 27879 ts_tablet_manager.cc:1909] T c27c4dfead60411d8806dd8789759d57 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:24.691408 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet c27c4dfead60411d8806dd8789759d57 (table rebalancer_dynamic_table_new_name [id=b615071706664c5a8eeb7a798e00d7ac]) successfully deleted
I20240502 01:54:24.693766 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet c27c4dfead60411d8806dd8789759d57 (table rebalancer_dynamic_table_new_name [id=b615071706664c5a8eeb7a798e00d7ac]) successfully deleted
I20240502 01:54:24.751597 21713 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:24.760830 21837 tablet_service.cc:1467] Processing CreateTablet for tablet 1033b9774fdd459bb67cfef4788d9ca7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cfa1f33cc3a645e791afb14fa37448ac]), partition=
I20240502 01:54:24.761265 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1033b9774fdd459bb67cfef4788d9ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:24.761619 24846 tablet_service.cc:1467] Processing CreateTablet for tablet 1033b9774fdd459bb67cfef4788d9ca7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cfa1f33cc3a645e791afb14fa37448ac]), partition=
I20240502 01:54:24.761940 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1033b9774fdd459bb67cfef4788d9ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:24.764726 24979 tablet_service.cc:1467] Processing CreateTablet for tablet 1033b9774fdd459bb67cfef4788d9ca7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cfa1f33cc3a645e791afb14fa37448ac]), partition=
I20240502 01:54:24.765195 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1033b9774fdd459bb67cfef4788d9ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:24.773269 21713 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:24.785200 27896 tablet_bootstrap.cc:492] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:24.788980 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 2036e577d94242728ae4d00fe845121b (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=64a3018c59994b83b4283de8ca4e1f27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:24.789454 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2036e577d94242728ae4d00fe845121b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:24.789758 27896 tablet_bootstrap.cc:654] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:24.794603 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 2036e577d94242728ae4d00fe845121b (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=64a3018c59994b83b4283de8ca4e1f27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:24.795024 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2036e577d94242728ae4d00fe845121b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:24.794991 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 2036e577d94242728ae4d00fe845121b (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=64a3018c59994b83b4283de8ca4e1f27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:24.795374 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2036e577d94242728ae4d00fe845121b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:24.795531 27901 tablet_bootstrap.cc:492] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:24.795670 24848 tablet_service.cc:1467] Processing CreateTablet for tablet dea30e2e1ef6473f880a001e57f27171 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=64a3018c59994b83b4283de8ca4e1f27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:24.796070 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea30e2e1ef6473f880a001e57f27171. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:24.799284 27901 tablet_bootstrap.cc:654] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:24.811264 27900 tablet_bootstrap.cc:492] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:24.813508 22098 tablet_service.cc:1467] Processing CreateTablet for tablet dea30e2e1ef6473f880a001e57f27171 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=64a3018c59994b83b4283de8ca4e1f27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:24.814095 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea30e2e1ef6473f880a001e57f27171. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:24.816049 27900 tablet_bootstrap.cc:654] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:24.815977 22229 tablet_service.cc:1467] Processing CreateTablet for tablet dea30e2e1ef6473f880a001e57f27171 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=64a3018c59994b83b4283de8ca4e1f27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:24.816584 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea30e2e1ef6473f880a001e57f27171. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:24.820621 27905 tablet_bootstrap.cc:492] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:24.822465 24847 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6f1c51e76342429deb0ccbf4ad8ce2 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=64a3018c59994b83b4283de8ca4e1f27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:24.822578 27904 tablet_bootstrap.cc:492] T 2036e577d94242728ae4d00fe845121b P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:24.822932 24847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6f1c51e76342429deb0ccbf4ad8ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:24.824244 27904 tablet_bootstrap.cc:654] T 2036e577d94242728ae4d00fe845121b P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:24.827037 21967 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6f1c51e76342429deb0ccbf4ad8ce2 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=64a3018c59994b83b4283de8ca4e1f27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:24.827497 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6f1c51e76342429deb0ccbf4ad8ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:24.828704 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6f1c51e76342429deb0ccbf4ad8ce2 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=64a3018c59994b83b4283de8ca4e1f27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:24.829092 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6f1c51e76342429deb0ccbf4ad8ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:24.833359 27896 tablet_bootstrap.cc:492] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:24.833557 27896 ts_tablet_manager.cc:1367] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.003s
I20240502 01:54:24.834437 27896 raft_consensus.cc:348] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:24.835363 27896 raft_consensus.cc:374] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:24.835521 27896 raft_consensus.cc:729] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:24.835848 27896 consensus_queue.cc:260] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:24.836539 27896 ts_tablet_manager.cc:1398] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:54:24.837085 27896 tablet_bootstrap.cc:492] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:24.839221 27896 tablet_bootstrap.cc:654] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:24.843745 27896 tablet_bootstrap.cc:492] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:24.844357 27896 ts_tablet_manager.cc:1367] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240502 01:54:24.845284 27896 raft_consensus.cc:348] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:24.849608 27896 raft_consensus.cc:374] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:24.849912 27896 raft_consensus.cc:729] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:24.850394 27896 consensus_queue.cc:260] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:24.853055 27896 ts_tablet_manager.cc:1398] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 01:54:24.853494 27896 tablet_bootstrap.cc:492] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:24.856104 27910 tablet_bootstrap.cc:492] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:24.858512 27905 tablet_bootstrap.cc:654] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:24.858469 27896 tablet_bootstrap.cc:654] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:24.862577 27910 tablet_bootstrap.cc:654] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:24.864107 27909 tablet_bootstrap.cc:492] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:24.866919 27909 tablet_bootstrap.cc:654] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:24.867288 27901 tablet_bootstrap.cc:492] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:24.872046 27901 ts_tablet_manager.cc:1367] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.077s user 0.003s sys 0.006s
I20240502 01:54:24.872102 27900 tablet_bootstrap.cc:492] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:24.872552 27900 ts_tablet_manager.cc:1367] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.000s
I20240502 01:54:24.873335 27901 raft_consensus.cc:348] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:24.875363 27900 raft_consensus.cc:348] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:24.875674 27900 raft_consensus.cc:374] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:24.875829 27900 raft_consensus.cc:729] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:24.876243 27900 consensus_queue.cc:260] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:24.876401 27901 raft_consensus.cc:374] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:24.876475 27901 raft_consensus.cc:729] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:24.877099 27900 ts_tablet_manager.cc:1398] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:24.877733 27901 consensus_queue.cc:260] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:24.878867 27640 raft_consensus.cc:482] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:24.879057 27640 raft_consensus.cc:504] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:24.879956 27640 leader_election.cc:288] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:24.880092 27909 tablet_bootstrap.cc:492] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:24.880335 27909 ts_tablet_manager.cc:1367] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240502 01:54:24.881151 27909 raft_consensus.cc:348] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:24.881425 27909 raft_consensus.cc:374] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:24.881528 27909 raft_consensus.cc:729] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:24.881870 27909 consensus_queue.cc:260] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:24.882316 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1033b9774fdd459bb67cfef4788d9ca7" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:24.882566 24999 raft_consensus.cc:2454] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:24.882984 24804 leader_election.cc:302] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:24.883443 27901 ts_tablet_manager.cc:1398] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:54:24.883949 27909 ts_tablet_manager.cc:1398] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:24.884363 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1033b9774fdd459bb67cfef4788d9ca7" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:24.889312 21857 raft_consensus.cc:2454] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:24.893051 27901 tablet_bootstrap.cc:492] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:24.895669 27901 tablet_bootstrap.cc:654] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:24.902453 27640 raft_consensus.cc:2781] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:24.902588 27640 raft_consensus.cc:482] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:24.902673 27640 raft_consensus.cc:3037] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:24.925662 27640 raft_consensus.cc:504] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:24.927043 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1033b9774fdd459bb67cfef4788d9ca7" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:24.927287 21857 raft_consensus.cc:3037] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:24.929483 27901 tablet_bootstrap.cc:492] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:24.930013 27901 ts_tablet_manager.cc:1367] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.001s
I20240502 01:54:24.930508 21857 raft_consensus.cc:2454] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:24.931156 24801 leader_election.cc:302] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:24.931350 27901 raft_consensus.cc:348] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:24.932152 27901 raft_consensus.cc:374] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:24.932390 27901 raft_consensus.cc:729] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:24.932965 27901 consensus_queue.cc:260] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:24.938881 27901 ts_tablet_manager.cc:1398] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 01:54:24.940409 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1033b9774fdd459bb67cfef4788d9ca7" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:24.940634 24999 raft_consensus.cc:3037] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:24.944285 24999 raft_consensus.cc:2454] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:24.945657 27640 leader_election.cc:288] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:24.946048 27640 raft_consensus.cc:2781] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:24.948221 27640 raft_consensus.cc:686] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:24.948624 27640 consensus_queue.cc:237] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:24.970280 27901 tablet_bootstrap.cc:492] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:24.971985 27901 tablet_bootstrap.cc:654] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:24.983913 27901 tablet_bootstrap.cc:492] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:24.984196 27901 ts_tablet_manager.cc:1367] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.001s
I20240502 01:54:24.985033 27901 raft_consensus.cc:348] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:24.985320 27901 raft_consensus.cc:374] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:24.985409 27901 raft_consensus.cc:729] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:24.985679 27901 consensus_queue.cc:260] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:24.990034 27901 ts_tablet_manager.cc:1398] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:54:24.993695 27904 tablet_bootstrap.cc:492] T 2036e577d94242728ae4d00fe845121b P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:24.995276 27904 ts_tablet_manager.cc:1367] T 2036e577d94242728ae4d00fe845121b P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.171s user 0.027s sys 0.022s
I20240502 01:54:24.996369 27904 raft_consensus.cc:348] T 2036e577d94242728ae4d00fe845121b P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:24.996666 27904 raft_consensus.cc:374] T 2036e577d94242728ae4d00fe845121b P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:24.996778 27904 raft_consensus.cc:729] T 2036e577d94242728ae4d00fe845121b P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:24.998848 27896 tablet_bootstrap.cc:492] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:24.999043 27896 ts_tablet_manager.cc:1367] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.146s user 0.017s sys 0.023s
I20240502 01:54:25.000286 27896 raft_consensus.cc:348] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:25.002080 27904 consensus_queue.cc:260] T 2036e577d94242728ae4d00fe845121b P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:25.000538 27896 raft_consensus.cc:374] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:25.002876 27896 raft_consensus.cc:729] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:25.003090 27910 tablet_bootstrap.cc:492] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:25.003335 27910 ts_tablet_manager.cc:1367] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.147s user 0.009s sys 0.024s
I20240502 01:54:25.003432 27904 ts_tablet_manager.cc:1398] T 2036e577d94242728ae4d00fe845121b P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 01:54:25.003759 27896 consensus_queue.cc:260] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
W20240502 01:54:25.000648 24802 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.198:42375, user_credentials={real_user=slave}} blocked reactor thread for 86524.9us
I20240502 01:54:25.004191 27910 raft_consensus.cc:348] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:25.004568 27910 raft_consensus.cc:374] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:25.004700 27910 raft_consensus.cc:729] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:25.005187 27904 tablet_bootstrap.cc:492] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:25.004752 21712 catalog_manager.cc:5484] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:25.006127 27896 ts_tablet_manager.cc:1398] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 01:54:25.007149 27910 consensus_queue.cc:260] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:25.007830 27910 ts_tablet_manager.cc:1398] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:25.008458 27904 tablet_bootstrap.cc:654] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:25.019157 27905 tablet_bootstrap.cc:492] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:25.019457 27905 ts_tablet_manager.cc:1367] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.199s user 0.008s sys 0.000s
I20240502 01:54:25.019536 27904 tablet_bootstrap.cc:492] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:25.019750 27904 ts_tablet_manager.cc:1367] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.010s
I20240502 01:54:25.020519 27905 raft_consensus.cc:348] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:25.020592 27904 raft_consensus.cc:348] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:25.020812 27905 raft_consensus.cc:374] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:25.020840 27904 raft_consensus.cc:374] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:25.020968 27905 raft_consensus.cc:729] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:25.020990 27904 raft_consensus.cc:729] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:25.021245 27904 consensus_queue.cc:260] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:25.021263 27905 consensus_queue.cc:260] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:25.021975 27904 ts_tablet_manager.cc:1398] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:25.022926 27905 ts_tablet_manager.cc:1398] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:25.037909 21857 consensus_queue.cc:237] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:25.040431 22119 raft_consensus.cc:1261] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:25.040483 24999 raft_consensus.cc:1261] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:25.040994 27516 consensus_queue.cc:1035] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:25.041307 27785 consensus_queue.cc:1035] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:25.044342 27516 raft_consensus.cc:2932] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:25.044994 22119 raft_consensus.cc:2932] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:25.051705 21714 catalog_manager.cc:5484] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:25.052166 27518 raft_consensus.cc:482] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:25.052311 27518 raft_consensus.cc:504] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:25.052649 24999 raft_consensus.cc:2932] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:25.053280 27518 leader_election.cc:288] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:25.055408 24865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f1c51e76342429deb0ccbf4ad8ce2" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:25.055703 24865 raft_consensus.cc:2454] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:25.056577 24583 tablet_service.cc:1514] Processing DeleteTablet for tablet e00feee2662a48d8a1a57f9e52275154 with delete_type TABLET_DATA_TOMBSTONED (TS 0c64d40b5d7644bd8b29feaa80eab72f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53832
I20240502 01:54:25.056963 27880 tablet_replica.cc:330] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:54:25.057237 27880 raft_consensus.cc:2229] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:25.057459 27880 raft_consensus.cc:2258] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:25.057696 21923 leader_election.cc:302] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:25.057982 27518 raft_consensus.cc:2781] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:25.058082 27518 raft_consensus.cc:482] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:25.058156 27518 raft_consensus.cc:3037] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:25.058146 27880 ts_tablet_manager.cc:1875] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:25.058105 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f1c51e76342429deb0ccbf4ad8ce2" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:25.058310 21857 raft_consensus.cc:2454] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:25.061981 27518 raft_consensus.cc:504] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:25.062907 27518 leader_election.cc:288] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:25.063395 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f1c51e76342429deb0ccbf4ad8ce2" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:25.063624 21857 raft_consensus.cc:3037] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:25.066928 27880 ts_tablet_manager.cc:1888] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:54:25.067595 24865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f1c51e76342429deb0ccbf4ad8ce2" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:25.067800 24865 raft_consensus.cc:3037] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:25.069680 21857 raft_consensus.cc:2454] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:25.070111 21922 leader_election.cc:302] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:25.070701 27518 raft_consensus.cc:2781] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:25.070683 24865 raft_consensus.cc:2454] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:25.070832 27518 raft_consensus.cc:686] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:54:25.071110 27518 consensus_queue.cc:237] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:25.071277 27880 log.cc:1196] T e00feee2662a48d8a1a57f9e52275154 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/e00feee2662a48d8a1a57f9e52275154
I20240502 01:54:25.072291 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet e00feee2662a48d8a1a57f9e52275154 (table rebalancer_extra_table_8 [id=de52a7578d964d1ba0e315c9f1e99b21]) successfully deleted
I20240502 01:54:25.077932 21714 catalog_manager.cc:5484] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:25.136727 21714 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:25.143509 21857 raft_consensus.cc:1261] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:25.143576 24999 raft_consensus.cc:1261] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:25.144157 27650 consensus_queue.cc:1035] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:25.144623 27640 consensus_queue.cc:1035] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:25.150355 21714 catalog_manager.cc:5576] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785) does not have the latest schema for tablet 1033b9774fdd459bb67cfef4788d9ca7 (table rebalancer_dynamic_table [id=cfa1f33cc3a645e791afb14fa37448ac]). Expected version 1 got 0
I20240502 01:54:25.161279 27947 tablet.cc:1722] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:25.162143 27947 tablet.cc:1620] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77: MemRowSet was empty: no flush needed.
I20240502 01:54:25.163869 21714 catalog_manager.cc:5576] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785) does not have the latest schema for tablet 1033b9774fdd459bb67cfef4788d9ca7 (table rebalancer_dynamic_table [id=cfa1f33cc3a645e791afb14fa37448ac]). Expected version 1 got 0
I20240502 01:54:25.164238 27946 tablet.cc:1722] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:25.165033 27946 tablet.cc:1620] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:25.169134 21714 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 1033b9774fdd459bb67cfef4788d9ca7 (table rebalancer_dynamic_table [id=cfa1f33cc3a645e791afb14fa37448ac]). Expected version 1 got 0
I20240502 01:54:25.180145 21713 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=cfa1f33cc3a645e791afb14fa37448ac] alter complete (version 1)
I20240502 01:54:25.185719 27951 tablet.cc:1722] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:25.186548 27951 tablet.cc:1620] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:25.187911 21714 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 1033b9774fdd459bb67cfef4788d9ca7 (table rebalancer_dynamic_table [id=cfa1f33cc3a645e791afb14fa37448ac]). Expected version 1 got 0
I20240502 01:54:25.282790 27795 raft_consensus.cc:482] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:25.283000 27795 raft_consensus.cc:504] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:25.283689 27795 leader_election.cc:288] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:25.286342 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2036e577d94242728ae4d00fe845121b" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:25.286589 21857 raft_consensus.cc:2454] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:25.287935 24536 leader_election.cc:302] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:25.288257 27795 raft_consensus.cc:2781] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:25.288347 27795 raft_consensus.cc:482] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:25.288436 27795 raft_consensus.cc:3037] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:25.289124 21713 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:25.291503 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2036e577d94242728ae4d00fe845121b" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:54:25.291791 22119 raft_consensus.cc:2454] T 2036e577d94242728ae4d00fe845121b P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:25.296949 27795 raft_consensus.cc:504] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:25.301385 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2036e577d94242728ae4d00fe845121b" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:54:25.301616 22118 raft_consensus.cc:3037] T 2036e577d94242728ae4d00fe845121b P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:25.304082 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2036e577d94242728ae4d00fe845121b" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:25.304296 21856 raft_consensus.cc:3037] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:25.304576 22118 raft_consensus.cc:2454] T 2036e577d94242728ae4d00fe845121b P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:25.305253 24538 leader_election.cc:302] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:54:25.305585 27954 raft_consensus.cc:2781] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:25.307044 21856 raft_consensus.cc:2454] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:25.308717 27946 tablet.cc:1722] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:25.309233 27946 tablet.cc:1620] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:25.312021 27951 tablet.cc:1722] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:25.312556 27951 tablet.cc:1620] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:25.314083 27947 tablet.cc:1722] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:25.314567 27947 tablet.cc:1620] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77: MemRowSet was empty: no flush needed.
I20240502 01:54:25.333565 27795 leader_election.cc:288] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:25.334729 27640 raft_consensus.cc:482] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:25.334909 27640 raft_consensus.cc:504] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:25.335777 27640 leader_election.cc:288] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:25.337013 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea30e2e1ef6473f880a001e57f27171" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:25.337162 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea30e2e1ef6473f880a001e57f27171" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:54:25.337265 22250 raft_consensus.cc:2454] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:25.337452 22119 raft_consensus.cc:2454] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:25.338152 24803 leader_election.cc:302] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:25.339454 27640 raft_consensus.cc:2781] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:25.339622 27640 raft_consensus.cc:482] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:25.339722 27640 raft_consensus.cc:3037] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:25.340309 21712 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=cfa1f33cc3a645e791afb14fa37448ac] alter complete (version 2)
I20240502 01:54:25.348438 27640 raft_consensus.cc:504] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:25.350945 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea30e2e1ef6473f880a001e57f27171" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:54:25.351217 22118 raft_consensus.cc:3037] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:25.354745 22118 raft_consensus.cc:2454] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:25.355742 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea30e2e1ef6473f880a001e57f27171" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:25.356062 22250 raft_consensus.cc:3037] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:25.359645 22250 raft_consensus.cc:2454] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:25.361667 24803 leader_election.cc:302] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:25.362143 27650 raft_consensus.cc:2781] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:25.362308 27650 raft_consensus.cc:686] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:25.362677 27650 consensus_queue.cc:237] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:25.363018 27954 raft_consensus.cc:686] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:54:25.363528 27954 consensus_queue.cc:237] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:25.364360 27640 leader_election.cc:288] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:25.367197 21712 catalog_manager.cc:5484] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:25.416930 21713 catalog_manager.cc:5484] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:25.442849 21707 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:25.453616 27946 tablet.cc:1722] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:25.457523 27951 tablet.cc:1722] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:25.459062 27947 tablet.cc:1722] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:25.481706 21707 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=cfa1f33cc3a645e791afb14fa37448ac] alter complete (version 3)
I20240502 01:54:25.492450 27761 consensus_queue.cc:1035] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:25.503262 27761 consensus_queue.cc:1035] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:25.594426 21712 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:25.594730 21712 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:25.601344 21712 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 1033b9774fdd459bb67cfef4788d9ca7
I20240502 01:54:25.602352 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 1033b9774fdd459bb67cfef4788d9ca7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:25 UTC) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:25.603000 28004 tablet_replica.cc:330] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:25.602907 24846 tablet_service.cc:1514] Processing DeleteTablet for tablet 1033b9774fdd459bb67cfef4788d9ca7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:25 UTC) from {username='slave'} at 127.0.0.1:51592
I20240502 01:54:25.603235 28004 raft_consensus.cc:2229] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:25.603451 28004 raft_consensus.cc:2258] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:25.604418 24979 tablet_service.cc:1514] Processing DeleteTablet for tablet 1033b9774fdd459bb67cfef4788d9ca7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:25 UTC) from {username='slave'} at 127.0.0.1:39964
I20240502 01:54:25.605407 28006 tablet_replica.cc:330] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:54:25.606909 28006 raft_consensus.cc:2229] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:25.607203 28006 raft_consensus.cc:2258] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:25.610196 28006 ts_tablet_manager.cc:1875] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:25.623888 28006 ts_tablet_manager.cc:1888] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:25.624075 28006 log.cc:1196] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/1033b9774fdd459bb67cfef4788d9ca7
I20240502 01:54:25.624598 28006 ts_tablet_manager.cc:1909] T 1033b9774fdd459bb67cfef4788d9ca7 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting consensus metadata
I20240502 01:54:25.629492 28004 ts_tablet_manager.cc:1875] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:25.636041 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet 1033b9774fdd459bb67cfef4788d9ca7 (table rebalancer_dynamic_table_new_name [id=cfa1f33cc3a645e791afb14fa37448ac]) successfully deleted
I20240502 01:54:25.639537 24583 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:25.640264 24713 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:25.662215 28005 tablet_replica.cc:330] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08: stopping tablet replica
I20240502 01:54:25.662497 28005 raft_consensus.cc:2229] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:25.663089 28005 raft_consensus.cc:2258] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:25.664022 28005 ts_tablet_manager.cc:1875] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:25.664474 21836 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:25.677975 28004 ts_tablet_manager.cc:1888] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:25.684540 24979 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:54:25.693579 28004 log.cc:1196] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/1033b9774fdd459bb67cfef4788d9ca7
I20240502 01:54:25.694892 28004 ts_tablet_manager.cc:1909] T 1033b9774fdd459bb67cfef4788d9ca7 P 759257ea7bef415ca698e60d82199d77: Deleting consensus metadata
I20240502 01:54:25.699499 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 1033b9774fdd459bb67cfef4788d9ca7 (table rebalancer_dynamic_table_new_name [id=cfa1f33cc3a645e791afb14fa37448ac]) successfully deleted
I20240502 01:54:25.702315 28005 ts_tablet_manager.cc:1888] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:25.702585 28005 log.cc:1196] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/wals/1033b9774fdd459bb67cfef4788d9ca7
I20240502 01:54:25.703330 28005 ts_tablet_manager.cc:1909] T 1033b9774fdd459bb67cfef4788d9ca7 P d7449bb1df214bf4a5f93e4631312f08: Deleting consensus metadata
I20240502 01:54:25.704424 21967 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:25.704476 21712 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:25.705205 21700 catalog_manager.cc:4830] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): tablet 1033b9774fdd459bb67cfef4788d9ca7 (table rebalancer_dynamic_table_new_name [id=cfa1f33cc3a645e791afb14fa37448ac]) successfully deleted
I20240502 01:54:25.711961 22229 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:54:25.715925 24713 tablet_service.cc:1467] Processing CreateTablet for tablet 7f1b82bb787e4b26ac181e4cd34486a3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=81cb4be424634fb19ceeb0593b2c0226]), partition=
I20240502 01:54:25.716393 24713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f1b82bb787e4b26ac181e4cd34486a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:25.717415 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 7f1b82bb787e4b26ac181e4cd34486a3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=81cb4be424634fb19ceeb0593b2c0226]), partition=
I20240502 01:54:25.717435 24979 tablet_service.cc:1467] Processing CreateTablet for tablet 7f1b82bb787e4b26ac181e4cd34486a3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=81cb4be424634fb19ceeb0593b2c0226]), partition=
I20240502 01:54:25.717813 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f1b82bb787e4b26ac181e4cd34486a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:25.717813 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f1b82bb787e4b26ac181e4cd34486a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:25.752338 28019 tablet_bootstrap.cc:492] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:25.752797 22100 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:25.757894 28019 tablet_bootstrap.cc:654] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:25.758095 28020 tablet_bootstrap.cc:492] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:25.759935 28020 tablet_bootstrap.cc:654] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:25.772789 28019 tablet_bootstrap.cc:492] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:25.773074 28019 ts_tablet_manager.cc:1367] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.000s
I20240502 01:54:25.774051 28019 raft_consensus.cc:348] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:25.774297 28019 raft_consensus.cc:374] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:25.774410 28019 raft_consensus.cc:729] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:25.774909 28019 consensus_queue.cc:260] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:25.775266 27779 raft_consensus.cc:482] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:25.775417 27779 raft_consensus.cc:504] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:25.776400 27779 leader_election.cc:288] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:25.776732 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f1b82bb787e4b26ac181e4cd34486a3" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
W20240502 01:54:25.777361 24671 leader_election.cc:341] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:25.777562 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f1b82bb787e4b26ac181e4cd34486a3" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:25.778154 28019 ts_tablet_manager.cc:1398] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20240502 01:54:25.778250 24669 leader_election.cc:341] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:25.778393 24669 leader_election.cc:302] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28157bc3e678428fb5f93c077d528513; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:25.778960 27779 raft_consensus.cc:2726] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:25.793157 28018 tablet_bootstrap.cc:492] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:25.795158 28018 tablet_bootstrap.cc:654] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:25.804275 27650 consensus_queue.cc:1035] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:25.830408 27650 consensus_queue.cc:1035] T dea30e2e1ef6473f880a001e57f27171 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:25.837309 28020 tablet_bootstrap.cc:492] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:25.839910 28020 ts_tablet_manager.cc:1367] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.082s user 0.003s sys 0.006s
I20240502 01:54:25.841195 28020 raft_consensus.cc:348] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:25.841542 28020 raft_consensus.cc:374] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:25.841725 28020 raft_consensus.cc:729] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:25.842777 27954 consensus_queue.cc:1035] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:25.843307 28020 consensus_queue.cc:260] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:25.844686 28020 ts_tablet_manager.cc:1398] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:54:25.869707 27976 consensus_queue.cc:1035] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:25.884167 24846 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:25.914932 28018 tablet_bootstrap.cc:492] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:25.915191 28018 ts_tablet_manager.cc:1367] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.122s user 0.010s sys 0.005s
I20240502 01:54:25.916061 28018 raft_consensus.cc:348] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:25.916388 28018 raft_consensus.cc:374] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:25.916525 28018 raft_consensus.cc:729] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:25.916812 28018 consensus_queue.cc:260] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:25.925050 28018 ts_tablet_manager.cc:1398] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240502 01:54:25.973634 27976 raft_consensus.cc:482] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:25.973834 27976 raft_consensus.cc:504] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:25.974690 27976 leader_election.cc:288] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:25.975802 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f1b82bb787e4b26ac181e4cd34486a3" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:25.976047 24999 raft_consensus.cc:2454] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:25.976442 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f1b82bb787e4b26ac181e4cd34486a3" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:25.976701 24732 raft_consensus.cc:2454] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:25.977686 24537 leader_election.cc:302] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:54:25.978432 27976 raft_consensus.cc:2781] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:25.978528 27976 raft_consensus.cc:482] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:25.978618 27976 raft_consensus.cc:3037] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:25.983271 27976 raft_consensus.cc:504] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:25.985183 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f1b82bb787e4b26ac181e4cd34486a3" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:25.985402 24999 raft_consensus.cc:3037] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:25.990047 24999 raft_consensus.cc:2454] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:25.990609 24539 leader_election.cc:302] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:25.990988 27977 raft_consensus.cc:2781] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:25.991048 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f1b82bb787e4b26ac181e4cd34486a3" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:25.991161 27977 raft_consensus.cc:686] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:54:25.991325 24732 raft_consensus.cc:3037] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:25.991523 27977 consensus_queue.cc:237] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:25.994038 21712 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:25.996649 24732 raft_consensus.cc:2454] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:26.000468 27976 leader_election.cc:288] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:26.013402 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 9b297d603934468a907a1e1e08b9c276 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=75c3f22aa26848b68c2a3c2b5a6a5b3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:26.013913 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b297d603934468a907a1e1e08b9c276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.013916 24981 tablet_service.cc:1467] Processing CreateTablet for tablet ae04b73f8a574264a26a647400a44235 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=75c3f22aa26848b68c2a3c2b5a6a5b3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:26.014375 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae04b73f8a574264a26a647400a44235. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.016224 22229 tablet_service.cc:1467] Processing CreateTablet for tablet c0877e287cb94abf98723449b9805f80 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=75c3f22aa26848b68c2a3c2b5a6a5b3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:26.016634 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0877e287cb94abf98723449b9805f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.018031 21836 tablet_service.cc:1467] Processing CreateTablet for tablet ae04b73f8a574264a26a647400a44235 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=75c3f22aa26848b68c2a3c2b5a6a5b3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:26.018472 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae04b73f8a574264a26a647400a44235. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.018396 24979 tablet_service.cc:1467] Processing CreateTablet for tablet 9b297d603934468a907a1e1e08b9c276 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=75c3f22aa26848b68c2a3c2b5a6a5b3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:26.020037 21713 catalog_manager.cc:5484] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:26.020332 24582 tablet_service.cc:1467] Processing CreateTablet for tablet c0877e287cb94abf98723449b9805f80 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=75c3f22aa26848b68c2a3c2b5a6a5b3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:26.020742 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0877e287cb94abf98723449b9805f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.021262 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 9b297d603934468a907a1e1e08b9c276 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=75c3f22aa26848b68c2a3c2b5a6a5b3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:26.021656 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b297d603934468a907a1e1e08b9c276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.022207 21837 tablet_service.cc:1467] Processing CreateTablet for tablet c0877e287cb94abf98723449b9805f80 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=75c3f22aa26848b68c2a3c2b5a6a5b3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:26.022539 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0877e287cb94abf98723449b9805f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.023674 21967 tablet_service.cc:1467] Processing CreateTablet for tablet ae04b73f8a574264a26a647400a44235 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=75c3f22aa26848b68c2a3c2b5a6a5b3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:26.024078 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae04b73f8a574264a26a647400a44235. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.031095 28035 tablet_bootstrap.cc:492] T ae04b73f8a574264a26a647400a44235 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:26.035627 28035 tablet_bootstrap.cc:654] T ae04b73f8a574264a26a647400a44235 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.036645 28018 tablet_bootstrap.cc:492] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:26.038514 28018 tablet_bootstrap.cc:654] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.040607 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b297d603934468a907a1e1e08b9c276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.045536 28035 tablet_bootstrap.cc:492] T ae04b73f8a574264a26a647400a44235 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:26.046613 28035 ts_tablet_manager.cc:1367] T ae04b73f8a574264a26a647400a44235 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240502 01:54:26.048189 28020 tablet_bootstrap.cc:492] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:26.048117 28035 raft_consensus.cc:348] T ae04b73f8a574264a26a647400a44235 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:26.049882 28036 tablet_bootstrap.cc:492] T 9b297d603934468a907a1e1e08b9c276 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:26.050076 28035 raft_consensus.cc:374] T ae04b73f8a574264a26a647400a44235 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:26.050319 28035 raft_consensus.cc:729] T ae04b73f8a574264a26a647400a44235 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:26.050434 28020 tablet_bootstrap.cc:654] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.050565 28035 consensus_queue.cc:260] T ae04b73f8a574264a26a647400a44235 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:26.058115 28020 tablet_bootstrap.cc:492] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:26.058396 28020 ts_tablet_manager.cc:1367] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20240502 01:54:26.059242 28020 raft_consensus.cc:348] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:26.060271 28020 raft_consensus.cc:374] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:26.060420 28020 raft_consensus.cc:729] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:26.060770 28020 consensus_queue.cc:260] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:26.062536 28020 ts_tablet_manager.cc:1398] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:26.062958 28020 tablet_bootstrap.cc:492] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:26.066040 28018 tablet_bootstrap.cc:492] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:26.066311 28018 ts_tablet_manager.cc:1367] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.003s
I20240502 01:54:26.066538 28020 tablet_bootstrap.cc:654] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.071287 28035 ts_tablet_manager.cc:1398] T ae04b73f8a574264a26a647400a44235 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240502 01:54:26.071305 28034 tablet_bootstrap.cc:492] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:26.072926 28034 tablet_bootstrap.cc:654] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.073112 28036 tablet_bootstrap.cc:654] T 9b297d603934468a907a1e1e08b9c276 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.087551 28018 raft_consensus.cc:348] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:26.087997 28018 raft_consensus.cc:374] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:26.088094 28018 raft_consensus.cc:729] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:26.088429 28018 consensus_queue.cc:260] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:26.090127 28018 ts_tablet_manager.cc:1398] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240502 01:54:26.093729 28018 tablet_bootstrap.cc:492] T 9b297d603934468a907a1e1e08b9c276 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:26.098078 28018 tablet_bootstrap.cc:654] T 9b297d603934468a907a1e1e08b9c276 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.099493 28039 tablet_bootstrap.cc:492] T c0877e287cb94abf98723449b9805f80 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:26.100982 28039 tablet_bootstrap.cc:654] T c0877e287cb94abf98723449b9805f80 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.103144 28020 tablet_bootstrap.cc:492] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:26.103384 28039 tablet_bootstrap.cc:492] T c0877e287cb94abf98723449b9805f80 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:26.103569 28039 ts_tablet_manager.cc:1367] T c0877e287cb94abf98723449b9805f80 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:54:26.104223 28020 ts_tablet_manager.cc:1367] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.015s
I20240502 01:54:26.105161 28020 raft_consensus.cc:348] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:26.105440 28020 raft_consensus.cc:374] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:26.105542 28020 raft_consensus.cc:729] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:26.105679 28039 raft_consensus.cc:348] T c0877e287cb94abf98723449b9805f80 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:26.105789 28020 consensus_queue.cc:260] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:26.106485 28020 ts_tablet_manager.cc:1398] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:26.107833 28018 tablet_bootstrap.cc:492] T 9b297d603934468a907a1e1e08b9c276 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:26.108050 28018 ts_tablet_manager.cc:1367] T 9b297d603934468a907a1e1e08b9c276 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240502 01:54:26.109835 28018 raft_consensus.cc:348] T 9b297d603934468a907a1e1e08b9c276 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:26.110064 28018 raft_consensus.cc:374] T 9b297d603934468a907a1e1e08b9c276 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:26.110198 28018 raft_consensus.cc:729] T 9b297d603934468a907a1e1e08b9c276 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:26.110769 28018 consensus_queue.cc:260] T 9b297d603934468a907a1e1e08b9c276 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:26.114589 28018 ts_tablet_manager.cc:1398] T 9b297d603934468a907a1e1e08b9c276 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:54:26.114744 28039 raft_consensus.cc:374] T c0877e287cb94abf98723449b9805f80 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:26.114948 28039 raft_consensus.cc:729] T c0877e287cb94abf98723449b9805f80 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:26.115805 28034 tablet_bootstrap.cc:492] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:26.116072 28036 tablet_bootstrap.cc:492] T 9b297d603934468a907a1e1e08b9c276 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:26.116263 28036 ts_tablet_manager.cc:1367] T 9b297d603934468a907a1e1e08b9c276 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.007s
I20240502 01:54:26.116289 28034 ts_tablet_manager.cc:1367] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.000s
I20240502 01:54:26.117166 28036 raft_consensus.cc:348] T 9b297d603934468a907a1e1e08b9c276 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:26.117381 28039 consensus_queue.cc:260] T c0877e287cb94abf98723449b9805f80 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:26.116828 28034 raft_consensus.cc:348] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:26.118271 28034 raft_consensus.cc:374] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:26.118381 28034 raft_consensus.cc:729] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:26.118631 28034 consensus_queue.cc:260] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:26.119271 28034 ts_tablet_manager.cc:1398] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:26.120659 28034 tablet_bootstrap.cc:492] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:26.122151 28039 ts_tablet_manager.cc:1398] T c0877e287cb94abf98723449b9805f80 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240502 01:54:26.123165 28034 tablet_bootstrap.cc:654] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.126121 28036 raft_consensus.cc:374] T 9b297d603934468a907a1e1e08b9c276 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:26.126219 28036 raft_consensus.cc:729] T 9b297d603934468a907a1e1e08b9c276 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:26.126427 28036 consensus_queue.cc:260] T 9b297d603934468a907a1e1e08b9c276 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:26.126958 28036 ts_tablet_manager.cc:1398] T 9b297d603934468a907a1e1e08b9c276 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:54:26.134198 28034 tablet_bootstrap.cc:492] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:26.134382 28034 ts_tablet_manager.cc:1367] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20240502 01:54:26.135282 28034 raft_consensus.cc:348] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:26.135486 28034 raft_consensus.cc:374] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:26.135538 28034 raft_consensus.cc:729] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:26.135797 28034 consensus_queue.cc:260] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:26.136478 28034 ts_tablet_manager.cc:1398] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:54:26.144032 21712 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:26.151196 24999 raft_consensus.cc:1261] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:26.151683 24732 raft_consensus.cc:1261] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:26.151782 27977 consensus_queue.cc:1035] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:26.152417 27977 consensus_queue.cc:1035] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:26.160982 27516 raft_consensus.cc:482] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:26.161124 27516 raft_consensus.cc:504] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:26.161931 27516 leader_election.cc:288] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:26.162410 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae04b73f8a574264a26a647400a44235" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:54:26.162647 21988 raft_consensus.cc:2454] T ae04b73f8a574264a26a647400a44235 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:54:26.163065 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae04b73f8a574264a26a647400a44235" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:26.163355 25000 raft_consensus.cc:2454] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:54:26.163378 21790 leader_election.cc:302] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:26.164292 27516 raft_consensus.cc:2781] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:26.164384 27516 raft_consensus.cc:482] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:26.164454 27516 raft_consensus.cc:3037] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:26.167188 27516 raft_consensus.cc:504] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:26.169076 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae04b73f8a574264a26a647400a44235" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:54:26.169322 21988 raft_consensus.cc:3037] T ae04b73f8a574264a26a647400a44235 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:26.172035 21988 raft_consensus.cc:2454] T ae04b73f8a574264a26a647400a44235 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:54:26.172418 21790 leader_election.cc:302] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:26.173691 28062 raft_consensus.cc:2781] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:26.173856 28062 raft_consensus.cc:686] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:54:26.176246 27516 leader_election.cc:288] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:26.176954 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae04b73f8a574264a26a647400a44235" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:26.177254 25000 raft_consensus.cc:3037] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:26.183218 25000 raft_consensus.cc:2454] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:54:26.185540 21712 catalog_manager.cc:5576] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375) does not have the latest schema for tablet 7f1b82bb787e4b26ac181e4cd34486a3 (table rebalancer_dynamic_table [id=81cb4be424634fb19ceeb0593b2c0226]). Expected version 1 got 0
I20240502 01:54:26.184908 28062 consensus_queue.cc:237] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:26.186810 28067 tablet.cc:1722] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:26.192719 28067 tablet.cc:1620] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:26.197387 27774 raft_consensus.cc:482] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:26.197624 27774 raft_consensus.cc:504] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:26.198523 27774 leader_election.cc:288] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:26.199470 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b297d603934468a907a1e1e08b9c276" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:54:26.199736 22118 raft_consensus.cc:2454] T 9b297d603934468a907a1e1e08b9c276 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:26.200474 24936 leader_election.cc:302] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:54:26.200771 27774 raft_consensus.cc:2781] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:26.200911 27774 raft_consensus.cc:482] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:26.201022 27774 raft_consensus.cc:3037] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:26.205456 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b297d603934468a907a1e1e08b9c276" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:26.205761 24603 raft_consensus.cc:2454] T 9b297d603934468a907a1e1e08b9c276 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 0.
I20240502 01:54:26.204849 27774 raft_consensus.cc:504] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:26.207561 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b297d603934468a907a1e1e08b9c276" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:54:26.207828 22118 raft_consensus.cc:3037] T 9b297d603934468a907a1e1e08b9c276 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:26.212083 22118 raft_consensus.cc:2454] T 9b297d603934468a907a1e1e08b9c276 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:26.212723 24936 leader_election.cc:302] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:54:26.213505 27989 raft_consensus.cc:2781] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:26.218282 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b297d603934468a907a1e1e08b9c276" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:26.218567 24602 raft_consensus.cc:3037] T 9b297d603934468a907a1e1e08b9c276 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:26.219765 21707 catalog_manager.cc:5576] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375) does not have the latest schema for tablet 7f1b82bb787e4b26ac181e4cd34486a3 (table rebalancer_dynamic_table [id=81cb4be424634fb19ceeb0593b2c0226]). Expected version 1 got 0
I20240502 01:54:26.221952 24602 raft_consensus.cc:2454] T 9b297d603934468a907a1e1e08b9c276 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:54:26.223338 27774 leader_election.cc:288] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:26.227933 28068 tablet.cc:1722] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:26.229542 28068 tablet.cc:1620] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:26.235697 27989 raft_consensus.cc:686] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:54:26.236187 27989 consensus_queue.cc:237] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:26.243583 21707 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 7f1b82bb787e4b26ac181e4cd34486a3 (table rebalancer_dynamic_table [id=81cb4be424634fb19ceeb0593b2c0226]). Expected version 1 got 0
I20240502 01:54:26.254822 27977 raft_consensus.cc:482] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:26.255018 27977 raft_consensus.cc:504] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:26.255846 27977 leader_election.cc:288] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:26.256318 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0877e287cb94abf98723449b9805f80" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:26.256604 21856 raft_consensus.cc:2454] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:26.257318 24536 leader_election.cc:302] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:26.257637 27977 raft_consensus.cc:2781] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:26.257733 27977 raft_consensus.cc:482] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:26.257833 27977 raft_consensus.cc:3037] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:26.260026 21707 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=81cb4be424634fb19ceeb0593b2c0226] alter complete (version 1)
I20240502 01:54:26.260686 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0877e287cb94abf98723449b9805f80" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:26.261018 22249 raft_consensus.cc:2454] T c0877e287cb94abf98723449b9805f80 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:26.264645 27977 raft_consensus.cc:504] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:26.265782 28076 tablet.cc:1722] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:26.265935 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0877e287cb94abf98723449b9805f80" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:26.266150 21856 raft_consensus.cc:3037] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:26.269111 21856 raft_consensus.cc:2454] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:26.269532 21707 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 7f1b82bb787e4b26ac181e4cd34486a3 (table rebalancer_dynamic_table [id=81cb4be424634fb19ceeb0593b2c0226]). Expected version 1 got 0
I20240502 01:54:26.270015 24536 leader_election.cc:302] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:26.269888 21707 catalog_manager.cc:5484] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:26.270426 27976 raft_consensus.cc:2781] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:26.270625 27976 raft_consensus.cc:686] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:54:26.270941 27976 consensus_queue.cc:237] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:26.271106 21712 catalog_manager.cc:5484] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:26.271981 28076 tablet.cc:1620] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:26.274890 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0877e287cb94abf98723449b9805f80" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:26.275157 22249 raft_consensus.cc:3037] T c0877e287cb94abf98723449b9805f80 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:26.292644 21712 catalog_manager.cc:5484] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:26.293653 22249 raft_consensus.cc:2454] T c0877e287cb94abf98723449b9805f80 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:26.306195 27977 leader_election.cc:288] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:26.328671 22117 consensus_queue.cc:237] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:26.338783 25000 raft_consensus.cc:1261] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:26.338826 21856 raft_consensus.cc:1261] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:26.339452 27207 consensus_queue.cc:1035] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:26.339764 28053 consensus_queue.cc:1035] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:26.359860 27207 raft_consensus.cc:2932] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:26.361775 24999 raft_consensus.cc:2932] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:26.364948 21712 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:26.365814 21856 raft_consensus.cc:2932] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:26.373477 21711 catalog_manager.cc:5484] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:26.383554 28067 tablet.cc:1722] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:26.384094 28067 tablet.cc:1620] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:26.386088 28076 tablet.cc:1722] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:26.386593 28076 tablet.cc:1620] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:26.393908 28068 tablet.cc:1722] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:26.394405 28068 tablet.cc:1620] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:26.395006 22229 tablet_service.cc:1514] Processing DeleteTablet for tablet daaadb773202454894890c3379c4a58d with delete_type TABLET_DATA_TOMBSTONED (TS c2a28baa7ee2469294d94b7fe5455cff not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:26.412475 28085 tablet_replica.cc:330] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:26.412724 28085 raft_consensus.cc:2229] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:26.413023 28085 raft_consensus.cc:2258] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:26.416230 21712 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=81cb4be424634fb19ceeb0593b2c0226] alter complete (version 2)
I20240502 01:54:26.417527 28085 ts_tablet_manager.cc:1875] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:26.424116 28085 ts_tablet_manager.cc:1888] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:54:26.424240 28085 log.cc:1196] T daaadb773202454894890c3379c4a58d P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/daaadb773202454894890c3379c4a58d
I20240502 01:54:26.425119 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet daaadb773202454894890c3379c4a58d (table rebalance_test_table_15 [id=855af8637536470bb3f01aeac50e07e8]) successfully deleted
I20240502 01:54:26.519423 21713 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:26.529996 28067 tablet.cc:1722] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:26.530695 28076 tablet.cc:1722] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:26.534036 28068 tablet.cc:1722] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:26.549959 21713 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=81cb4be424634fb19ceeb0593b2c0226] alter complete (version 3)
I20240502 01:54:26.571399 28062 consensus_queue.cc:1035] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:26.602344 28062 consensus_queue.cc:1035] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:26.652071 21712 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:26.652348 21712 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:26.656359 21712 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 7f1b82bb787e4b26ac181e4cd34486a3
I20240502 01:54:26.657581 24979 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f1b82bb787e4b26ac181e4cd34486a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:26 UTC) from {username='slave'} at 127.0.0.1:39964
I20240502 01:54:26.658228 28140 tablet_replica.cc:330] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:54:26.658382 24713 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f1b82bb787e4b26ac181e4cd34486a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:26 UTC) from {username='slave'} at 127.0.0.1:41650
I20240502 01:54:26.658402 24582 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f1b82bb787e4b26ac181e4cd34486a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:26 UTC) from {username='slave'} at 127.0.0.1:53832
I20240502 01:54:26.658469 28140 raft_consensus.cc:2229] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:26.658739 28140 raft_consensus.cc:2258] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:26.659456 28141 tablet_replica.cc:330] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:54:26.659660 28140 ts_tablet_manager.cc:1875] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:26.659816 28141 raft_consensus.cc:2229] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:26.660173 28141 raft_consensus.cc:2258] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:26.661584 28141 ts_tablet_manager.cc:1875] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:26.667029 28140 ts_tablet_manager.cc:1888] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:26.667184 28140 log.cc:1196] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/7f1b82bb787e4b26ac181e4cd34486a3
I20240502 01:54:26.667805 28140 ts_tablet_manager.cc:1909] T 7f1b82bb787e4b26ac181e4cd34486a3 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting consensus metadata
I20240502 01:54:26.669207 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet 7f1b82bb787e4b26ac181e4cd34486a3 (table rebalancer_dynamic_table_new_name [id=81cb4be424634fb19ceeb0593b2c0226]) successfully deleted
I20240502 01:54:26.669739 28141 ts_tablet_manager.cc:1888] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:26.669865 28141 log.cc:1196] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/7f1b82bb787e4b26ac181e4cd34486a3
I20240502 01:54:26.670677 28141 ts_tablet_manager.cc:1909] T 7f1b82bb787e4b26ac181e4cd34486a3 P 28157bc3e678428fb5f93c077d528513: Deleting consensus metadata
I20240502 01:54:26.672932 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet 7f1b82bb787e4b26ac181e4cd34486a3 (table rebalancer_dynamic_table_new_name [id=81cb4be424634fb19ceeb0593b2c0226]) successfully deleted
I20240502 01:54:26.677268 28142 tablet_replica.cc:330] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:54:26.677484 28142 raft_consensus.cc:2229] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:26.677830 28142 raft_consensus.cc:2258] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:26.678607 28142 ts_tablet_manager.cc:1875] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:26.686823 28142 ts_tablet_manager.cc:1888] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:26.687108 28142 log.cc:1196] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/7f1b82bb787e4b26ac181e4cd34486a3
I20240502 01:54:26.687721 28142 ts_tablet_manager.cc:1909] T 7f1b82bb787e4b26ac181e4cd34486a3 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting consensus metadata
I20240502 01:54:26.689479 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet 7f1b82bb787e4b26ac181e4cd34486a3 (table rebalancer_dynamic_table_new_name [id=81cb4be424634fb19ceeb0593b2c0226]) successfully deleted
I20240502 01:54:26.701344 28074 consensus_queue.cc:1035] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:26.703001 27954 consensus_queue.cc:1035] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:26.728915 27976 consensus_queue.cc:1035] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:26.739356 27989 consensus_queue.cc:1035] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:26.759543 21712 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:26.773869 21967 tablet_service.cc:1467] Processing CreateTablet for tablet ab4e5cf725534c5aa190225623b5493a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ef54dd1093d54b889cec89e071c57d4b]), partition=
I20240502 01:54:26.774257 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab4e5cf725534c5aa190225623b5493a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.778401 24846 tablet_service.cc:1467] Processing CreateTablet for tablet ab4e5cf725534c5aa190225623b5493a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ef54dd1093d54b889cec89e071c57d4b]), partition=
I20240502 01:54:26.778828 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab4e5cf725534c5aa190225623b5493a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.787887 22100 tablet_service.cc:1467] Processing CreateTablet for tablet ab4e5cf725534c5aa190225623b5493a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ef54dd1093d54b889cec89e071c57d4b]), partition=
I20240502 01:54:26.788276 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab4e5cf725534c5aa190225623b5493a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.804914 28167 tablet_bootstrap.cc:492] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:26.805501 28168 tablet_bootstrap.cc:492] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:26.806547 28167 tablet_bootstrap.cc:654] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.806939 28168 tablet_bootstrap.cc:654] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.808828 28170 tablet_bootstrap.cc:492] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:26.815877 28170 tablet_bootstrap.cc:654] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.820183 28167 tablet_bootstrap.cc:492] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:26.820492 28167 ts_tablet_manager.cc:1367] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.009s
I20240502 01:54:26.821537 28167 raft_consensus.cc:348] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:26.822003 28167 raft_consensus.cc:374] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:26.822121 28167 raft_consensus.cc:729] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:26.822507 28167 consensus_queue.cc:260] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:26.830780 28167 ts_tablet_manager.cc:1398] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:54:26.833417 28168 tablet_bootstrap.cc:492] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:26.833871 28168 ts_tablet_manager.cc:1367] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.009s
I20240502 01:54:26.834753 28168 raft_consensus.cc:348] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:26.835871 28168 raft_consensus.cc:374] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:26.836086 28168 raft_consensus.cc:729] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:26.836501 28168 consensus_queue.cc:260] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:26.837543 28168 ts_tablet_manager.cc:1398] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:54:26.854739 28170 tablet_bootstrap.cc:492] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:26.857046 21712 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:26.861943 28170 ts_tablet_manager.cc:1367] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.005s
I20240502 01:54:26.865044 28170 raft_consensus.cc:348] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:26.865494 28170 raft_consensus.cc:374] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:26.865656 28170 raft_consensus.cc:729] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:26.866082 28170 consensus_queue.cc:260] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:26.869179 28170 ts_tablet_manager.cc:1398] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 01:54:26.872526 22229 tablet_service.cc:1467] Processing CreateTablet for tablet 36d358202b544e74bcc378060082df4d (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=048399628e3c4ff99f3a5d3a67b8ff83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:26.872526 21837 tablet_service.cc:1467] Processing CreateTablet for tablet 36d358202b544e74bcc378060082df4d (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=048399628e3c4ff99f3a5d3a67b8ff83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:26.872723 24846 tablet_service.cc:1467] Processing CreateTablet for tablet 36d358202b544e74bcc378060082df4d (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=048399628e3c4ff99f3a5d3a67b8ff83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:26.872992 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36d358202b544e74bcc378060082df4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.872992 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36d358202b544e74bcc378060082df4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.873158 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36d358202b544e74bcc378060082df4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.878959 24847 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad2f9c27d284098a8e4f20ebc4b3d8c (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=048399628e3c4ff99f3a5d3a67b8ff83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:26.879387 24847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad2f9c27d284098a8e4f20ebc4b3d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.880333 22231 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad2f9c27d284098a8e4f20ebc4b3d8c (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=048399628e3c4ff99f3a5d3a67b8ff83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:26.880705 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad2f9c27d284098a8e4f20ebc4b3d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.882305 24713 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad2f9c27d284098a8e4f20ebc4b3d8c (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=048399628e3c4ff99f3a5d3a67b8ff83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:26.882768 24713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad2f9c27d284098a8e4f20ebc4b3d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.888995 28170 tablet_bootstrap.cc:492] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:26.890969 28170 tablet_bootstrap.cc:654] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.892652 24979 tablet_service.cc:1467] Processing CreateTablet for tablet c7ebb7d204f04c60a32c450e703e4690 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=048399628e3c4ff99f3a5d3a67b8ff83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:26.893157 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7ebb7d204f04c60a32c450e703e4690. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.893395 22230 tablet_service.cc:1467] Processing CreateTablet for tablet c7ebb7d204f04c60a32c450e703e4690 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=048399628e3c4ff99f3a5d3a67b8ff83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:26.893800 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7ebb7d204f04c60a32c450e703e4690. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.897295 21836 tablet_service.cc:1467] Processing CreateTablet for tablet c7ebb7d204f04c60a32c450e703e4690 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=048399628e3c4ff99f3a5d3a67b8ff83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:26.897732 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7ebb7d204f04c60a32c450e703e4690. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.909569 28179 tablet_bootstrap.cc:492] T c7ebb7d204f04c60a32c450e703e4690 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:26.911370 28179 tablet_bootstrap.cc:654] T c7ebb7d204f04c60a32c450e703e4690 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.914438 28177 tablet_bootstrap.cc:492] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:26.916065 28177 tablet_bootstrap.cc:654] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.923132 28177 tablet_bootstrap.cc:492] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:26.924247 28177 ts_tablet_manager.cc:1367] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20240502 01:54:26.925329 28177 raft_consensus.cc:348] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:26.927211 28177 raft_consensus.cc:374] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:26.927636 28177 raft_consensus.cc:729] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:26.927673 28170 tablet_bootstrap.cc:492] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:26.928140 28177 consensus_queue.cc:260] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:26.929494 28170 ts_tablet_manager.cc:1367] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.003s
I20240502 01:54:26.929710 28177 ts_tablet_manager.cc:1398] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:54:26.930757 28177 tablet_bootstrap.cc:492] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:26.932101 28170 raft_consensus.cc:348] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:26.932651 28177 tablet_bootstrap.cc:654] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.932647 28170 raft_consensus.cc:374] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:26.932749 28170 raft_consensus.cc:729] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:26.934744 28170 consensus_queue.cc:260] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:26.937088 28170 ts_tablet_manager.cc:1398] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:54:26.937487 28170 tablet_bootstrap.cc:492] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:26.939314 28170 tablet_bootstrap.cc:654] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.945981 28177 tablet_bootstrap.cc:492] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:26.948596 28177 ts_tablet_manager.cc:1367] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20240502 01:54:26.951318 28177 raft_consensus.cc:348] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:26.952694 28177 raft_consensus.cc:374] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:26.953068 28177 raft_consensus.cc:729] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:26.953572 28177 consensus_queue.cc:260] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:26.955013 28177 ts_tablet_manager.cc:1398] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 01:54:26.956096 28177 tablet_bootstrap.cc:492] T c7ebb7d204f04c60a32c450e703e4690 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:26.956502 28176 tablet_bootstrap.cc:492] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:26.946609 28180 tablet_bootstrap.cc:492] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:26.958143 28177 tablet_bootstrap.cc:654] T c7ebb7d204f04c60a32c450e703e4690 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.965334 28179 tablet_bootstrap.cc:492] T c7ebb7d204f04c60a32c450e703e4690 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:26.965565 28179 ts_tablet_manager.cc:1367] T c7ebb7d204f04c60a32c450e703e4690 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.056s user 0.000s sys 0.017s
I20240502 01:54:26.966378 28179 raft_consensus.cc:348] T c7ebb7d204f04c60a32c450e703e4690 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:26.966650 28179 raft_consensus.cc:374] T c7ebb7d204f04c60a32c450e703e4690 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:26.966789 28179 raft_consensus.cc:729] T c7ebb7d204f04c60a32c450e703e4690 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:26.967060 28179 consensus_queue.cc:260] T c7ebb7d204f04c60a32c450e703e4690 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:26.969305 28180 tablet_bootstrap.cc:654] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.972496 28176 tablet_bootstrap.cc:654] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.972544 28179 ts_tablet_manager.cc:1398] T c7ebb7d204f04c60a32c450e703e4690 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 01:54:26.977458 28170 tablet_bootstrap.cc:492] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:26.977694 28170 ts_tablet_manager.cc:1367] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20240502 01:54:26.978605 28170 raft_consensus.cc:348] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:26.978883 28170 raft_consensus.cc:374] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:26.978993 28170 raft_consensus.cc:729] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:26.979279 28170 consensus_queue.cc:260] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:26.979794 28170 ts_tablet_manager.cc:1398] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:26.982860 28177 tablet_bootstrap.cc:492] T c7ebb7d204f04c60a32c450e703e4690 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:26.983090 28177 ts_tablet_manager.cc:1367] T c7ebb7d204f04c60a32c450e703e4690 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20240502 01:54:26.983897 28177 raft_consensus.cc:348] T c7ebb7d204f04c60a32c450e703e4690 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:26.984223 28177 raft_consensus.cc:374] T c7ebb7d204f04c60a32c450e703e4690 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:26.984347 28177 raft_consensus.cc:729] T c7ebb7d204f04c60a32c450e703e4690 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:26.984684 28177 consensus_queue.cc:260] T c7ebb7d204f04c60a32c450e703e4690 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:26.985469 28177 ts_tablet_manager.cc:1398] T c7ebb7d204f04c60a32c450e703e4690 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:27.009049 28180 tablet_bootstrap.cc:492] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:27.009297 28180 ts_tablet_manager.cc:1367] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.063s user 0.005s sys 0.001s
I20240502 01:54:27.010154 28180 raft_consensus.cc:348] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:27.012950 28180 raft_consensus.cc:374] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:27.013608 28180 raft_consensus.cc:729] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:27.013942 28180 consensus_queue.cc:260] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:27.016167 28180 ts_tablet_manager.cc:1398] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 01:54:27.018651 28176 tablet_bootstrap.cc:492] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:27.018869 28176 ts_tablet_manager.cc:1367] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.002s
I20240502 01:54:27.019790 28176 raft_consensus.cc:348] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:27.020077 28176 raft_consensus.cc:374] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:27.020195 28176 raft_consensus.cc:729] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:27.020496 28176 consensus_queue.cc:260] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:27.025930 28176 ts_tablet_manager.cc:1398] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 01:54:27.037158 28176 tablet_bootstrap.cc:492] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:27.041777 28176 tablet_bootstrap.cc:654] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:27.054111 28176 tablet_bootstrap.cc:492] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:27.054384 28176 ts_tablet_manager.cc:1367] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.001s
I20240502 01:54:27.055509 28176 raft_consensus.cc:348] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:27.055745 28176 raft_consensus.cc:374] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:27.055842 28176 raft_consensus.cc:729] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:27.056078 28176 consensus_queue.cc:260] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:27.056862 28176 ts_tablet_manager.cc:1398] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:27.065894 28023 raft_consensus.cc:482] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:27.066073 28023 raft_consensus.cc:504] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:27.066915 28023 leader_election.cc:288] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:27.067308 27916 raft_consensus.cc:482] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:27.067559 27916 raft_consensus.cc:504] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:27.067710 24865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad2f9c27d284098a8e4f20ebc4b3d8c" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:27.067979 24865 raft_consensus.cc:2454] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:27.068626 27916 leader_election.cc:288] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:27.069274 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab4e5cf725534c5aa190225623b5493a" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:54:27.069487 21988 raft_consensus.cc:2454] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:27.069424 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad2f9c27d284098a8e4f20ebc4b3d8c" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:27.069694 22249 raft_consensus.cc:2454] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:27.070313 24670 leader_election.cc:302] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:27.070727 28023 raft_consensus.cc:2781] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:27.070869 28023 raft_consensus.cc:482] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:27.070972 28023 raft_consensus.cc:3037] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:27.074366 22117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab4e5cf725534c5aa190225623b5493a" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:54:27.074656 22117 raft_consensus.cc:2454] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:27.075421 24801 leader_election.cc:302] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:27.076242 27650 raft_consensus.cc:2781] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:27.076390 27650 raft_consensus.cc:482] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:27.076494 27650 raft_consensus.cc:3037] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:27.079725 27650 raft_consensus.cc:504] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:27.085392 28062 raft_consensus.cc:482] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:27.085547 28062 raft_consensus.cc:504] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:27.086398 28062 leader_election.cc:288] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:27.087316 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ebb7d204f04c60a32c450e703e4690" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:27.087627 25000 raft_consensus.cc:2454] T c7ebb7d204f04c60a32c450e703e4690 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:54:27.090852 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ebb7d204f04c60a32c450e703e4690" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:27.091122 22249 raft_consensus.cc:2454] T c7ebb7d204f04c60a32c450e703e4690 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:54:27.091609 21792 leader_election.cc:302] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:27.093611 28064 raft_consensus.cc:2781] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:27.093734 28064 raft_consensus.cc:482] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:27.093818 28064 raft_consensus.cc:3037] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:27.098070 22117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab4e5cf725534c5aa190225623b5493a" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:54:27.098282 22117 raft_consensus.cc:3037] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:27.101521 22117 raft_consensus.cc:2454] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:27.102241 28023 raft_consensus.cc:504] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:27.102984 28023 leader_election.cc:288] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:27.103353 24803 leader_election.cc:302] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:27.104348 27916 raft_consensus.cc:2781] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:27.104538 27916 raft_consensus.cc:686] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:27.104844 27916 consensus_queue.cc:237] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:27.106021 24866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad2f9c27d284098a8e4f20ebc4b3d8c" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:27.099238 28064 raft_consensus.cc:504] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:27.107888 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad2f9c27d284098a8e4f20ebc4b3d8c" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:27.108101 22249 raft_consensus.cc:3037] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:27.108237 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ebb7d204f04c60a32c450e703e4690" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:27.108474 22250 raft_consensus.cc:3037] T c7ebb7d204f04c60a32c450e703e4690 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:27.111680 22250 raft_consensus.cc:2454] T c7ebb7d204f04c60a32c450e703e4690 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:54:27.112498 21792 leader_election.cc:302] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:27.112830 22249 raft_consensus.cc:2454] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:27.113306 24670 leader_election.cc:302] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:27.113395 28062 raft_consensus.cc:2781] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:27.113780 28023 raft_consensus.cc:2781] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:27.113991 28062 raft_consensus.cc:686] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:54:27.114333 28062 consensus_queue.cc:237] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:27.115254 28023 raft_consensus.cc:686] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:27.115983 28023 consensus_queue.cc:237] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:27.118574 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ebb7d204f04c60a32c450e703e4690" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:27.118888 25000 raft_consensus.cc:3037] T c7ebb7d204f04c60a32c450e703e4690 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:27.123680 24866 raft_consensus.cc:3037] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:27.123771 25000 raft_consensus.cc:2454] T c7ebb7d204f04c60a32c450e703e4690 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:54:27.128279 27650 leader_election.cc:288] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:27.129273 28064 leader_election.cc:288] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:27.130767 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab4e5cf725534c5aa190225623b5493a" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:54:27.130970 21988 raft_consensus.cc:3037] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:27.134434 21988 raft_consensus.cc:2454] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:27.134248 24866 raft_consensus.cc:2454] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:27.165362 21707 catalog_manager.cc:5484] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:27.171535 21713 catalog_manager.cc:5484] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:27.174432 21707 catalog_manager.cc:5484] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:27.204686 28199 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:27.204921 28199 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:27.205843 28199 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:27.206192 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:27.206439 22250 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:27.206722 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:27.206982 21856 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:27.207131 24803 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:27.207393 28199 raft_consensus.cc:2781] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:27.207513 28199 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:27.207619 28199 raft_consensus.cc:3037] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:27.210860 28199 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:27.211623 28199 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:27.211880 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:27.212074 21856 raft_consensus.cc:3037] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:27.214445 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:27.214712 22250 raft_consensus.cc:3037] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:27.215626 21856 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:27.216090 24801 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:27.216508 28199 raft_consensus.cc:2781] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:27.216837 28199 raft_consensus.cc:686] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:27.217175 28199 consensus_queue.cc:237] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:27.218708 22250 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:27.247594 21712 catalog_manager.cc:5484] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:27.266274 24582 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:27.287982 21712 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:27.292289 21967 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:27.345782 24713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:27.385977 21988 raft_consensus.cc:1261] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:27.387141 28199 consensus_queue.cc:1035] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:27.392925 22053 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.193:33785, user_credentials={real_user=slave}} blocked reactor thread for 75484.4us
I20240502 01:54:27.413600 22118 raft_consensus.cc:1261] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:27.414089 22230 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:54:27.416267 28199 consensus_queue.cc:1035] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:27.419507 21712 catalog_manager.cc:5576] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411) does not have the latest schema for tablet ab4e5cf725534c5aa190225623b5493a (table rebalancer_dynamic_table [id=ef54dd1093d54b889cec89e071c57d4b]). Expected version 1 got 0
I20240502 01:54:27.435952 28246 tablet.cc:1722] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:27.442904 21712 catalog_manager.cc:5576] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411) does not have the latest schema for tablet ab4e5cf725534c5aa190225623b5493a (table rebalancer_dynamic_table [id=ef54dd1093d54b889cec89e071c57d4b]). Expected version 1 got 0
I20240502 01:54:27.437705 28247 tablet.cc:1722] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:27.452749 21837 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:27.452772 28247 tablet.cc:1620] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412: MemRowSet was empty: no flush needed.
I20240502 01:54:27.461349 24846 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:27.440184 28246 tablet.cc:1620] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:27.474354 21712 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=ef54dd1093d54b889cec89e071c57d4b] alter complete (version 1)
I20240502 01:54:27.474870 24979 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:27.537317 21707 catalog_manager.cc:5576] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879) does not have the latest schema for tablet ab4e5cf725534c5aa190225623b5493a (table rebalancer_dynamic_table [id=ef54dd1093d54b889cec89e071c57d4b]). Expected version 1 got 0
I20240502 01:54:27.541391 22100 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:27.581382 28249 tablet.cc:1722] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:27.584304 21707 catalog_manager.cc:5576] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879) does not have the latest schema for tablet ab4e5cf725534c5aa190225623b5493a (table rebalancer_dynamic_table [id=ef54dd1093d54b889cec89e071c57d4b]). Expected version 1 got 0
I20240502 01:54:27.582156 28249 tablet.cc:1620] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a: MemRowSet was empty: no flush needed.
I20240502 01:54:27.587565 21707 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:27.596547 28208 consensus_queue.cc:1035] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:27.602546 28246 tablet.cc:1722] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:27.603080 28246 tablet.cc:1620] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08: MemRowSet was empty: no flush needed.
I20240502 01:54:27.604429 28247 tablet.cc:1722] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:27.604914 28247 tablet.cc:1620] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412: MemRowSet was empty: no flush needed.
I20240502 01:54:27.611531 28249 tablet.cc:1722] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:27.612628 21712 catalog_manager.cc:5576] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879) does not have the latest schema for tablet ab4e5cf725534c5aa190225623b5493a (table rebalancer_dynamic_table [id=ef54dd1093d54b889cec89e071c57d4b]). Expected version 2 got 1
I20240502 01:54:27.613922 28249 tablet.cc:1620] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a: MemRowSet was empty: no flush needed.
I20240502 01:54:27.619614 21713 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=ef54dd1093d54b889cec89e071c57d4b] alter complete (version 2)
I20240502 01:54:27.628072 28208 consensus_queue.cc:1035] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:27.631202 28064 consensus_queue.cc:1035] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:27.661115 28062 consensus_queue.cc:1035] T c7ebb7d204f04c60a32c450e703e4690 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:27.717962 28206 consensus_queue.cc:1035] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:27.723359 28206 consensus_queue.cc:1035] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:27.730933 21707 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:27.741259 28246 tablet.cc:1722] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:27.742512 28249 tablet.cc:1722] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:27.747923 28247 tablet.cc:1722] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:27.756592 21707 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=ef54dd1093d54b889cec89e071c57d4b] alter complete (version 3)
I20240502 01:54:27.768559 22117 consensus_queue.cc:237] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:27.770818 24603 raft_consensus.cc:1261] T 3cf315f0117444ae9c5390a1e19460d9 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:27.771107 21856 raft_consensus.cc:1261] T 3cf315f0117444ae9c5390a1e19460d9 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:27.771346 28234 consensus_queue.cc:1035] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:27.771764 28229 consensus_queue.cc:1035] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:27.775216 28229 raft_consensus.cc:2932] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:27.775925 24603 raft_consensus.cc:2932] T 3cf315f0117444ae9c5390a1e19460d9 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:27.776356 21857 raft_consensus.cc:2932] T 3cf315f0117444ae9c5390a1e19460d9 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:27.782176 21711 catalog_manager.cc:5484] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:27.788990 21967 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cf315f0117444ae9c5390a1e19460d9 with delete_type TABLET_DATA_TOMBSTONED (TS 06543b05a16f42099bc86db2a0284412 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:27.794745 21707 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:27.808145 21969 tablet_service.cc:1467] Processing CreateTablet for tablet 0b73e5695b28424ba127d1f434d1e88c (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=ca142102a7e44673b2cac39cee9b5bcd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:27.808750 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b73e5695b28424ba127d1f434d1e88c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.811676 28282 tablet_replica.cc:330] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:27.811851 28282 raft_consensus.cc:2229] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:27.812037 28282 raft_consensus.cc:2258] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:27.812700 28282 ts_tablet_manager.cc:1875] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:27.813306 24582 tablet_service.cc:1467] Processing CreateTablet for tablet 0b73e5695b28424ba127d1f434d1e88c (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=ca142102a7e44673b2cac39cee9b5bcd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:27.813769 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b73e5695b28424ba127d1f434d1e88c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.820245 24713 tablet_service.cc:1467] Processing CreateTablet for tablet 0b73e5695b28424ba127d1f434d1e88c (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=ca142102a7e44673b2cac39cee9b5bcd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:27.820863 24713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b73e5695b28424ba127d1f434d1e88c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.822412 28282 ts_tablet_manager.cc:1888] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:54:27.822526 28282 log.cc:1196] T 3cf315f0117444ae9c5390a1e19460d9 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/3cf315f0117444ae9c5390a1e19460d9
I20240502 01:54:27.823320 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 3cf315f0117444ae9c5390a1e19460d9 (table rebalance_test_table_1 [id=eaf29e0201fc4b738b1b30c13a773ffd]) successfully deleted
I20240502 01:54:27.827617 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 37827d24c61a465f88b679cd694df62f (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=ca142102a7e44673b2cac39cee9b5bcd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:27.828047 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37827d24c61a465f88b679cd694df62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.830461 24979 tablet_service.cc:1467] Processing CreateTablet for tablet 37827d24c61a465f88b679cd694df62f (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=ca142102a7e44673b2cac39cee9b5bcd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:27.830904 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37827d24c61a465f88b679cd694df62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.835258 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 3c81ce3a90f04b818186bb6e43259d13 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=ca142102a7e44673b2cac39cee9b5bcd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:27.835696 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c81ce3a90f04b818186bb6e43259d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.839301 21967 tablet_service.cc:1467] Processing CreateTablet for tablet 37827d24c61a465f88b679cd694df62f (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=ca142102a7e44673b2cac39cee9b5bcd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:27.839700 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37827d24c61a465f88b679cd694df62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.856508 21968 tablet_service.cc:1467] Processing CreateTablet for tablet 3c81ce3a90f04b818186bb6e43259d13 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=ca142102a7e44673b2cac39cee9b5bcd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:27.856990 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c81ce3a90f04b818186bb6e43259d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.859927 24981 tablet_service.cc:1467] Processing CreateTablet for tablet 3c81ce3a90f04b818186bb6e43259d13 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=ca142102a7e44673b2cac39cee9b5bcd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:27.860428 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c81ce3a90f04b818186bb6e43259d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.862417 21707 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:27.862643 21707 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:27.866618 28288 tablet_bootstrap.cc:492] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:27.869165 28288 tablet_bootstrap.cc:654] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:27.871145 21707 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet ab4e5cf725534c5aa190225623b5493a
I20240502 01:54:27.872740 21968 tablet_service.cc:1514] Processing DeleteTablet for tablet ab4e5cf725534c5aa190225623b5493a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:27 UTC) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:27.872768 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet ab4e5cf725534c5aa190225623b5493a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:27 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:27.873392 28291 tablet_replica.cc:330] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:27.873447 24846 tablet_service.cc:1514] Processing DeleteTablet for tablet ab4e5cf725534c5aa190225623b5493a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:27 UTC) from {username='slave'} at 127.0.0.1:51592
I20240502 01:54:27.873986 28289 tablet_bootstrap.cc:492] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:27.874099 28292 tablet_replica.cc:330] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08: stopping tablet replica
I20240502 01:54:27.874419 28292 raft_consensus.cc:2229] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:27.875025 28291 raft_consensus.cc:2229] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:27.875494 28292 raft_consensus.cc:2258] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:27.875995 28291 raft_consensus.cc:2258] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:27.877420 28292 ts_tablet_manager.cc:1875] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:27.878098 28291 ts_tablet_manager.cc:1875] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:27.878713 28282 tablet_replica.cc:330] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:27.879307 28282 raft_consensus.cc:2229] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:27.879503 28282 raft_consensus.cc:2258] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:27.881424 28282 ts_tablet_manager.cc:1875] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:27.888275 28289 tablet_bootstrap.cc:654] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:27.891412 28292 ts_tablet_manager.cc:1888] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:27.891541 28287 tablet_bootstrap.cc:492] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:27.893262 28287 tablet_bootstrap.cc:654] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:27.896850 28291 ts_tablet_manager.cc:1888] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:27.900441 28289 tablet_bootstrap.cc:492] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:27.900667 28289 ts_tablet_manager.cc:1367] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20240502 01:54:27.901473 28289 raft_consensus.cc:348] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:27.901820 28289 raft_consensus.cc:374] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:27.901916 28289 raft_consensus.cc:729] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:27.901930 28288 tablet_bootstrap.cc:492] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:27.902077 28288 ts_tablet_manager.cc:1367] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.000s
I20240502 01:54:27.902204 28289 consensus_queue.cc:260] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:27.902984 28289 ts_tablet_manager.cc:1398] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:27.903714 28289 tablet_bootstrap.cc:492] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:27.903894 28292 log.cc:1196] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/wals/ab4e5cf725534c5aa190225623b5493a
I20240502 01:54:27.904510 28292 ts_tablet_manager.cc:1909] T ab4e5cf725534c5aa190225623b5493a P d7449bb1df214bf4a5f93e4631312f08: Deleting consensus metadata
I20240502 01:54:27.904673 28291 log.cc:1196] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/ab4e5cf725534c5aa190225623b5493a
I20240502 01:54:27.904712 28282 ts_tablet_manager.cc:1888] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:27.905207 28291 ts_tablet_manager.cc:1909] T ab4e5cf725534c5aa190225623b5493a P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:27.905541 28288 raft_consensus.cc:348] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:27.906162 28288 raft_consensus.cc:374] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:27.906512 21700 catalog_manager.cc:4830] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): tablet ab4e5cf725534c5aa190225623b5493a (table rebalancer_dynamic_table_new_name [id=ef54dd1093d54b889cec89e071c57d4b]) successfully deleted
I20240502 01:54:27.906572 28288 raft_consensus.cc:729] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:27.907058 28282 log.cc:1196] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/ab4e5cf725534c5aa190225623b5493a
I20240502 01:54:27.907943 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet ab4e5cf725534c5aa190225623b5493a (table rebalancer_dynamic_table_new_name [id=ef54dd1093d54b889cec89e071c57d4b]) successfully deleted
I20240502 01:54:27.908296 28288 consensus_queue.cc:260] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:27.909340 28282 ts_tablet_manager.cc:1909] T ab4e5cf725534c5aa190225623b5493a P 06543b05a16f42099bc86db2a0284412: Deleting consensus metadata
I20240502 01:54:27.912333 28288 ts_tablet_manager.cc:1398] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:54:27.912494 28287 tablet_bootstrap.cc:492] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:27.912746 28287 ts_tablet_manager.cc:1367] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240502 01:54:27.913774 28287 raft_consensus.cc:348] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:27.914034 28287 raft_consensus.cc:374] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:27.905956 28289 tablet_bootstrap.cc:654] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:27.914698 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet ab4e5cf725534c5aa190225623b5493a (table rebalancer_dynamic_table_new_name [id=ef54dd1093d54b889cec89e071c57d4b]) successfully deleted
I20240502 01:54:27.915141 28288 tablet_bootstrap.cc:492] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:27.916262 28222 raft_consensus.cc:482] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:27.916389 28222 raft_consensus.cc:504] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:27.917057 28222 leader_election.cc:288] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:27.918591 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37827d24c61a465f88b679cd694df62f" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:27.921146 25000 raft_consensus.cc:2454] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:27.918978 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37827d24c61a465f88b679cd694df62f" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:27.923085 28287 raft_consensus.cc:729] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:27.923074 21925 leader_election.cc:302] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:27.923350 28222 raft_consensus.cc:2781] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:27.923425 28222 raft_consensus.cc:482] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:27.923503 28222 raft_consensus.cc:3037] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:27.924211 28287 consensus_queue.cc:260] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:27.925061 24735 raft_consensus.cc:2454] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:27.926319 28222 raft_consensus.cc:504] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:27.927367 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37827d24c61a465f88b679cd694df62f" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:27.927599 25000 raft_consensus.cc:3037] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:27.928189 28287 ts_tablet_manager.cc:1398] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 01:54:27.928687 28287 tablet_bootstrap.cc:492] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:27.930399 25000 raft_consensus.cc:2454] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:27.930832 21925 leader_election.cc:302] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:27.931161 27761 raft_consensus.cc:2781] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:27.931295 27761 raft_consensus.cc:686] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:54:27.931567 27761 consensus_queue.cc:237] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:27.931828 28293 tablet_bootstrap.cc:492] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:27.934686 28222 leader_election.cc:288] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:27.934878 28293 tablet_bootstrap.cc:654] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:27.939455 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37827d24c61a465f88b679cd694df62f" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:27.939689 24735 raft_consensus.cc:3037] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:27.943066 24735 raft_consensus.cc:2454] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:27.944924 28287 tablet_bootstrap.cc:654] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:27.946826 28289 tablet_bootstrap.cc:492] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:27.947026 28289 ts_tablet_manager.cc:1367] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20240502 01:54:27.947909 28289 raft_consensus.cc:348] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:27.948163 28289 raft_consensus.cc:374] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:27.948251 28289 raft_consensus.cc:729] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:27.916883 28288 tablet_bootstrap.cc:654] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:27.948562 28289 consensus_queue.cc:260] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:27.949242 28289 ts_tablet_manager.cc:1398] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:27.949522 28289 tablet_bootstrap.cc:492] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:27.949995 21713 catalog_manager.cc:5484] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:27.957648 28289 tablet_bootstrap.cc:654] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:27.960459 28288 tablet_bootstrap.cc:492] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:27.960652 28287 tablet_bootstrap.cc:492] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:27.960673 28288 ts_tablet_manager.cc:1367] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.046s user 0.001s sys 0.005s
I20240502 01:54:27.960922 28287 ts_tablet_manager.cc:1367] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.004s
I20240502 01:54:27.961465 28288 raft_consensus.cc:348] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:27.961773 28288 raft_consensus.cc:374] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:27.961912 28288 raft_consensus.cc:729] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:27.961827 28287 raft_consensus.cc:348] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:27.962249 28287 raft_consensus.cc:374] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:27.962401 28287 raft_consensus.cc:729] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:27.962445 28288 consensus_queue.cc:260] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:27.962733 28287 consensus_queue.cc:260] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:27.963380 28287 ts_tablet_manager.cc:1398] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:27.963510 28288 ts_tablet_manager.cc:1398] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:27.973310 28293 tablet_bootstrap.cc:492] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:27.974097 21711 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:27.978127 28289 tablet_bootstrap.cc:492] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:27.978340 28289 ts_tablet_manager.cc:1367] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20240502 01:54:27.979516 28289 raft_consensus.cc:348] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:27.979789 28289 raft_consensus.cc:374] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:27.979903 28289 raft_consensus.cc:729] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:27.980208 28289 consensus_queue.cc:260] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:27.981037 28293 ts_tablet_manager.cc:1367] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.049s user 0.005s sys 0.000s
I20240502 01:54:27.982517 28289 ts_tablet_manager.cc:1398] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:54:27.985137 28293 raft_consensus.cc:348] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:27.985440 28293 raft_consensus.cc:374] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:27.986943 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 70e53c1d20c44e439eaa609615fa7137 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c9c501f8f45f4abca84bb4722d438787]), partition=
I20240502 01:54:27.987293 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70e53c1d20c44e439eaa609615fa7137. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.987493 24582 tablet_service.cc:1467] Processing CreateTablet for tablet 70e53c1d20c44e439eaa609615fa7137 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c9c501f8f45f4abca84bb4722d438787]), partition=
I20240502 01:54:27.990119 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70e53c1d20c44e439eaa609615fa7137. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.991947 22230 tablet_service.cc:1467] Processing CreateTablet for tablet 70e53c1d20c44e439eaa609615fa7137 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c9c501f8f45f4abca84bb4722d438787]), partition=
I20240502 01:54:27.992311 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70e53c1d20c44e439eaa609615fa7137. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.985615 28293 raft_consensus.cc:729] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:28.013758 28293 consensus_queue.cc:260] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:28.023737 28293 ts_tablet_manager.cc:1398] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.042s user 0.003s sys 0.000s
I20240502 01:54:28.025338 28293 tablet_bootstrap.cc:492] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:28.033610 28307 tablet_bootstrap.cc:492] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:28.036116 28293 tablet_bootstrap.cc:654] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:28.036103 28307 tablet_bootstrap.cc:654] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:28.042057 28308 tablet_bootstrap.cc:492] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:28.046032 28307 tablet_bootstrap.cc:492] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:28.046180 28307 ts_tablet_manager.cc:1367] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240502 01:54:28.049118 28307 raft_consensus.cc:348] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:28.050503 28308 tablet_bootstrap.cc:654] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:28.050654 28307 raft_consensus.cc:374] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:28.050779 28307 raft_consensus.cc:729] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:28.052910 28307 consensus_queue.cc:260] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:28.054239 28307 ts_tablet_manager.cc:1398] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:54:28.060644 28293 tablet_bootstrap.cc:492] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:28.060842 28293 ts_tablet_manager.cc:1367] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.005s
I20240502 01:54:28.061681 28293 raft_consensus.cc:348] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:28.067651 28293 raft_consensus.cc:374] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:28.067819 28293 raft_consensus.cc:729] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:28.068184 28293 consensus_queue.cc:260] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:28.069031 28293 ts_tablet_manager.cc:1398] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:54:28.076481 27761 raft_consensus.cc:482] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:28.076624 27761 raft_consensus.cc:504] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:28.077463 27761 leader_election.cc:288] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:28.078692 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b73e5695b28424ba127d1f434d1e88c" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:28.078913 24603 raft_consensus.cc:2454] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:28.079401 21923 leader_election.cc:302] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 0c64d40b5d7644bd8b29feaa80eab72f; no voters:
I20240502 01:54:28.079905 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b73e5695b28424ba127d1f434d1e88c" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:28.080185 24735 raft_consensus.cc:2454] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:28.081046 27761 raft_consensus.cc:2781] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:28.081166 27761 raft_consensus.cc:482] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:28.081248 27761 raft_consensus.cc:3037] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:28.082875 28293 tablet_bootstrap.cc:492] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:28.084735 28293 tablet_bootstrap.cc:654] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:28.084733 27761 raft_consensus.cc:504] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:28.086619 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b73e5695b28424ba127d1f434d1e88c" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:28.086848 24603 raft_consensus.cc:3037] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:28.087455 28308 tablet_bootstrap.cc:492] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:28.087643 28308 ts_tablet_manager.cc:1367] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.011s
I20240502 01:54:28.087826 28293 tablet_bootstrap.cc:492] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:28.087998 28293 ts_tablet_manager.cc:1367] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:54:28.087913 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b73e5695b28424ba127d1f434d1e88c" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:28.088165 24735 raft_consensus.cc:3037] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:28.088325 28308 raft_consensus.cc:348] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:28.088573 28308 raft_consensus.cc:374] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:28.088632 28308 raft_consensus.cc:729] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:28.088864 28308 consensus_queue.cc:260] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:28.090211 24603 raft_consensus.cc:2454] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:28.090638 21923 leader_election.cc:302] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 0c64d40b5d7644bd8b29feaa80eab72f; no voters:
I20240502 01:54:28.090996 24735 raft_consensus.cc:2454] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:28.091240 28222 raft_consensus.cc:2781] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:28.091343 28222 raft_consensus.cc:686] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:54:28.091590 28222 consensus_queue.cc:237] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:28.093153 28308 ts_tablet_manager.cc:1398] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:54:28.095052 28293 raft_consensus.cc:348] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:28.095337 28293 raft_consensus.cc:374] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:28.097149 21707 catalog_manager.cc:5484] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:28.098719 28293 raft_consensus.cc:729] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:28.098992 28293 consensus_queue.cc:260] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:28.100309 28293 ts_tablet_manager.cc:1398] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 01:54:28.108656 27761 leader_election.cc:288] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:28.114557 27976 raft_consensus.cc:482] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:28.114755 27976 raft_consensus.cc:504] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:28.115418 27976 leader_election.cc:288] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:28.116387 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e53c1d20c44e439eaa609615fa7137" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:28.116624 22250 raft_consensus.cc:2454] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:28.117367 24538 leader_election.cc:302] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:28.117836 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e53c1d20c44e439eaa609615fa7137" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:54:28.118041 22120 raft_consensus.cc:2454] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 0.
I20240502 01:54:28.118372 27976 raft_consensus.cc:2781] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:28.118481 27976 raft_consensus.cc:482] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:28.118582 27976 raft_consensus.cc:3037] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:28.120786 27976 raft_consensus.cc:504] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:28.121904 27976 leader_election.cc:288] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:28.122325 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e53c1d20c44e439eaa609615fa7137" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:54:28.122365 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e53c1d20c44e439eaa609615fa7137" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:28.122529 22250 raft_consensus.cc:3037] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:28.122524 22120 raft_consensus.cc:3037] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:28.125340 22250 raft_consensus.cc:2454] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:28.125828 22120 raft_consensus.cc:2454] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:54:28.126113 24538 leader_election.cc:302] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:28.126387 27976 raft_consensus.cc:2781] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:28.126621 27976 raft_consensus.cc:686] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:54:28.126933 27976 consensus_queue.cc:237] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:28.131210 21713 catalog_manager.cc:5484] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:28.199426 27761 raft_consensus.cc:482] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:28.199605 27761 raft_consensus.cc:504] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:28.200270 27761 leader_election.cc:288] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:28.200858 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c81ce3a90f04b818186bb6e43259d13" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:28.201172 25000 raft_consensus.cc:2454] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:28.201531 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c81ce3a90f04b818186bb6e43259d13" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:28.201786 24603 raft_consensus.cc:2454] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:28.201790 21925 leader_election.cc:302] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:28.203881 27995 raft_consensus.cc:2781] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:28.203996 27995 raft_consensus.cc:482] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:28.204098 27995 raft_consensus.cc:3037] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:28.208398 27995 raft_consensus.cc:504] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:28.209471 27995 leader_election.cc:288] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:28.209514 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c81ce3a90f04b818186bb6e43259d13" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:28.209750 24603 raft_consensus.cc:3037] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:28.210016 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c81ce3a90f04b818186bb6e43259d13" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:28.210243 25000 raft_consensus.cc:3037] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:28.213161 24603 raft_consensus.cc:2454] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:28.214313 21923 leader_election.cc:302] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 0c64d40b5d7644bd8b29feaa80eab72f; no voters:
I20240502 01:54:28.214660 27995 raft_consensus.cc:2781] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:28.214967 25000 raft_consensus.cc:2454] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:28.215292 27995 raft_consensus.cc:686] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:54:28.215557 27995 consensus_queue.cc:237] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:28.237390 21711 catalog_manager.cc:5484] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:28.251433 21711 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:28.290558 22249 raft_consensus.cc:1261] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:28.291287 22120 raft_consensus.cc:1261] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:28.291960 27976 consensus_queue.cc:1035] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:28.292575 27977 consensus_queue.cc:1035] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:28.370429 21707 catalog_manager.cc:5576] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093) does not have the latest schema for tablet 70e53c1d20c44e439eaa609615fa7137 (table rebalancer_dynamic_table [id=c9c501f8f45f4abca84bb4722d438787]). Expected version 1 got 0
I20240502 01:54:28.373088 21707 catalog_manager.cc:5576] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879) does not have the latest schema for tablet 70e53c1d20c44e439eaa609615fa7137 (table rebalancer_dynamic_table [id=c9c501f8f45f4abca84bb4722d438787]). Expected version 1 got 0
I20240502 01:54:28.376689 28356 tablet.cc:1722] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:28.381412 28356 tablet.cc:1620] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a: MemRowSet was empty: no flush needed.
I20240502 01:54:28.382091 27761 consensus_queue.cc:1035] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:28.394352 21707 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=c9c501f8f45f4abca84bb4722d438787] alter complete (version 1)
I20240502 01:54:28.395515 28355 tablet.cc:1722] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:28.396376 28355 tablet.cc:1620] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff: MemRowSet was empty: no flush needed.
I20240502 01:54:28.401822 21707 catalog_manager.cc:5576] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093) does not have the latest schema for tablet 70e53c1d20c44e439eaa609615fa7137 (table rebalancer_dynamic_table [id=c9c501f8f45f4abca84bb4722d438787]). Expected version 1 got 0
I20240502 01:54:28.413697 28354 tablet.cc:1722] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:28.419642 28354 tablet.cc:1620] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:28.454136 28222 consensus_queue.cc:1035] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:28.505327 21714 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:28.522845 28356 tablet.cc:1722] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:28.522970 28354 tablet.cc:1722] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:28.523347 28356 tablet.cc:1620] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a: MemRowSet was empty: no flush needed.
I20240502 01:54:28.523524 28354 tablet.cc:1620] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:28.530405 28355 tablet.cc:1722] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:28.531152 28355 tablet.cc:1620] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff: MemRowSet was empty: no flush needed.
I20240502 01:54:28.544524 21707 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=c9c501f8f45f4abca84bb4722d438787] alter complete (version 2)
I20240502 01:54:28.562155 27995 consensus_queue.cc:1035] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:28.567919 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:54:28.602545 27995 consensus_queue.cc:1035] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:28.619037 27761 consensus_queue.cc:1035] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:28.635138 24582 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:28.643915 28222 consensus_queue.cc:1035] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:28.645386 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:28.652333 21707 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:28.671562 28354 tablet.cc:1722] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:28.674714 28356 tablet.cc:1722] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:28.677696 28355 tablet.cc:1722] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:28.691165 21714 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=c9c501f8f45f4abca84bb4722d438787] alter complete (version 3)
I20240502 01:54:28.701915 22230 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:54:28.710536 24715 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:28.729192 21837 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:28.741694 22100 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:28.776120 24846 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:54:28.794014 21715 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:28.799494 21711 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:28.799687 21711 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:28.808916 21711 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 70e53c1d20c44e439eaa609615fa7137
I20240502 01:54:28.810211 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 70e53c1d20c44e439eaa609615fa7137 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:28 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:28.812283 24582 tablet_service.cc:1514] Processing DeleteTablet for tablet 70e53c1d20c44e439eaa609615fa7137 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:28 UTC) from {username='slave'} at 127.0.0.1:53832
I20240502 01:54:28.812737 22230 tablet_service.cc:1514] Processing DeleteTablet for tablet 70e53c1d20c44e439eaa609615fa7137 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:28 UTC) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:28.818150 22098 tablet_service.cc:1467] Processing CreateTablet for tablet 5a82a89e92d243a78373077ea5a1f5de (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=41d42a6d1aa141c98532c43f5e0b1289]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:28.818629 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a82a89e92d243a78373077ea5a1f5de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:28.821280 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 5a82a89e92d243a78373077ea5a1f5de (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=41d42a6d1aa141c98532c43f5e0b1289]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:28.821874 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a82a89e92d243a78373077ea5a1f5de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:28.822542 24981 tablet_service.cc:1467] Processing CreateTablet for tablet 641ed1f2bf7e40d7a8a6472cf9cd5519 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=41d42a6d1aa141c98532c43f5e0b1289]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:28.822983 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 641ed1f2bf7e40d7a8a6472cf9cd5519. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:28.825997 21969 tablet_service.cc:1467] Processing CreateTablet for tablet 5a82a89e92d243a78373077ea5a1f5de (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=41d42a6d1aa141c98532c43f5e0b1289]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:28.826488 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a82a89e92d243a78373077ea5a1f5de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:28.829638 28387 tablet_replica.cc:330] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:28.829563 21837 tablet_service.cc:1467] Processing CreateTablet for tablet 641ed1f2bf7e40d7a8a6472cf9cd5519 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=41d42a6d1aa141c98532c43f5e0b1289]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:28.829874 28387 raft_consensus.cc:2229] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:28.829967 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 641ed1f2bf7e40d7a8a6472cf9cd5519. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:28.830086 28387 raft_consensus.cc:2258] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:28.832379 28387 ts_tablet_manager.cc:1875] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:28.845856 24846 tablet_service.cc:1467] Processing CreateTablet for tablet 641ed1f2bf7e40d7a8a6472cf9cd5519 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=41d42a6d1aa141c98532c43f5e0b1289]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:28.846288 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 641ed1f2bf7e40d7a8a6472cf9cd5519. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:28.858194 24583 tablet_service.cc:1467] Processing CreateTablet for tablet ba18c000cd3f42978ed8fd01a4ded60c (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=41d42a6d1aa141c98532c43f5e0b1289]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:28.858702 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba18c000cd3f42978ed8fd01a4ded60c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:28.862051 22229 tablet_service.cc:1467] Processing CreateTablet for tablet ba18c000cd3f42978ed8fd01a4ded60c (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=41d42a6d1aa141c98532c43f5e0b1289]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:28.862483 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba18c000cd3f42978ed8fd01a4ded60c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:28.862555 22100 tablet_service.cc:1467] Processing CreateTablet for tablet ba18c000cd3f42978ed8fd01a4ded60c (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=41d42a6d1aa141c98532c43f5e0b1289]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:28.863096 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba18c000cd3f42978ed8fd01a4ded60c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:28.863270 28390 tablet_bootstrap.cc:492] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:28.865490 28391 tablet_bootstrap.cc:492] T 5a82a89e92d243a78373077ea5a1f5de P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:28.867441 28391 tablet_bootstrap.cc:654] T 5a82a89e92d243a78373077ea5a1f5de P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:28.868973 28389 tablet_replica.cc:330] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:28.869277 28389 raft_consensus.cc:2229] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:28.869596 28389 raft_consensus.cc:2258] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:28.871014 28389 ts_tablet_manager.cc:1875] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:28.847000 28387 ts_tablet_manager.cc:1888] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:28.873772 28387 log.cc:1196] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/70e53c1d20c44e439eaa609615fa7137
I20240502 01:54:28.875681 28387 ts_tablet_manager.cc:1909] T 70e53c1d20c44e439eaa609615fa7137 P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:28.878414 28388 tablet_replica.cc:330] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:54:28.878712 28388 raft_consensus.cc:2229] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:28.881332 28388 raft_consensus.cc:2258] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:28.882381 28388 ts_tablet_manager.cc:1875] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:28.876816 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 70e53c1d20c44e439eaa609615fa7137 (table rebalancer_dynamic_table_new_name [id=c9c501f8f45f4abca84bb4722d438787]) successfully deleted
I20240502 01:54:28.887522 28390 tablet_bootstrap.cc:654] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:28.888552 28394 tablet_bootstrap.cc:492] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:28.889323 28396 tablet_bootstrap.cc:492] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:28.891691 28396 tablet_bootstrap.cc:654] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:28.893376 28392 tablet_bootstrap.cc:492] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:28.894852 28392 tablet_bootstrap.cc:654] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:28.896253 28397 tablet_bootstrap.cc:492] T 5a82a89e92d243a78373077ea5a1f5de P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:28.900835 28397 tablet_bootstrap.cc:654] T 5a82a89e92d243a78373077ea5a1f5de P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:28.908610 28390 tablet_bootstrap.cc:492] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:28.908855 28390 ts_tablet_manager.cc:1367] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20240502 01:54:28.910337 28390 raft_consensus.cc:348] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:28.910792 28390 raft_consensus.cc:374] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:28.910944 28390 raft_consensus.cc:729] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:28.911949 21711 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:28.911319 28390 consensus_queue.cc:260] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:28.913262 28390 ts_tablet_manager.cc:1398] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:54:28.918319 28393 tablet_bootstrap.cc:492] T ba18c000cd3f42978ed8fd01a4ded60c P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:28.920590 28397 tablet_bootstrap.cc:492] T 5a82a89e92d243a78373077ea5a1f5de P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:28.920778 28397 ts_tablet_manager.cc:1367] T 5a82a89e92d243a78373077ea5a1f5de P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.002s
I20240502 01:54:28.921697 28392 tablet_bootstrap.cc:492] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:28.921887 28392 ts_tablet_manager.cc:1367] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.008s
I20240502 01:54:28.922627 28396 tablet_bootstrap.cc:492] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:28.922791 28396 ts_tablet_manager.cc:1367] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.003s
I20240502 01:54:28.922999 28392 raft_consensus.cc:348] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:28.923522 28392 raft_consensus.cc:374] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:28.923657 28392 raft_consensus.cc:729] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:28.924533 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 791fbda0957e456082495e420d222d6b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8ba57952df5f43c5be003c4b1c675036]), partition=
I20240502 01:54:28.925004 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 791fbda0957e456082495e420d222d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:28.925477 24981 tablet_service.cc:1467] Processing CreateTablet for tablet 791fbda0957e456082495e420d222d6b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8ba57952df5f43c5be003c4b1c675036]), partition=
I20240502 01:54:28.925894 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 791fbda0957e456082495e420d222d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:28.926096 28397 raft_consensus.cc:348] T 5a82a89e92d243a78373077ea5a1f5de P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:28.926349 28397 raft_consensus.cc:374] T 5a82a89e92d243a78373077ea5a1f5de P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:28.926456 28397 raft_consensus.cc:729] T 5a82a89e92d243a78373077ea5a1f5de P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:28.926748 28397 consensus_queue.cc:260] T 5a82a89e92d243a78373077ea5a1f5de P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:28.927373 28397 ts_tablet_manager.cc:1398] T 5a82a89e92d243a78373077ea5a1f5de P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:28.923979 28392 consensus_queue.cc:260] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:28.932217 28392 ts_tablet_manager.cc:1398] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:54:28.936270 28391 tablet_bootstrap.cc:492] T 5a82a89e92d243a78373077ea5a1f5de P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:28.936470 28391 ts_tablet_manager.cc:1367] T 5a82a89e92d243a78373077ea5a1f5de P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.000s
I20240502 01:54:28.937247 28391 raft_consensus.cc:348] T 5a82a89e92d243a78373077ea5a1f5de P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:28.937398 28395 tablet_bootstrap.cc:492] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:28.937839 28391 raft_consensus.cc:374] T 5a82a89e92d243a78373077ea5a1f5de P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:28.938009 28391 raft_consensus.cc:729] T 5a82a89e92d243a78373077ea5a1f5de P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:28.938036 28394 tablet_bootstrap.cc:654] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:28.940079 28391 consensus_queue.cc:260] T 5a82a89e92d243a78373077ea5a1f5de P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:28.947973 28391 ts_tablet_manager.cc:1398] T 5a82a89e92d243a78373077ea5a1f5de P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:54:28.951233 28393 tablet_bootstrap.cc:654] T ba18c000cd3f42978ed8fd01a4ded60c P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:28.925575 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 791fbda0957e456082495e420d222d6b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8ba57952df5f43c5be003c4b1c675036]), partition=
I20240502 01:54:28.952009 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 791fbda0957e456082495e420d222d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:28.963084 28390 tablet_bootstrap.cc:492] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:28.963755 28391 tablet_bootstrap.cc:492] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:28.968669 28395 tablet_bootstrap.cc:654] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:28.970933 28396 raft_consensus.cc:348] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:28.972324 28393 tablet_bootstrap.cc:492] T ba18c000cd3f42978ed8fd01a4ded60c P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:28.972566 28393 ts_tablet_manager.cc:1367] T ba18c000cd3f42978ed8fd01a4ded60c P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.000s
I20240502 01:54:28.973975 28393 raft_consensus.cc:348] T ba18c000cd3f42978ed8fd01a4ded60c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:28.974397 28393 raft_consensus.cc:374] T ba18c000cd3f42978ed8fd01a4ded60c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:28.974778 28393 raft_consensus.cc:729] T ba18c000cd3f42978ed8fd01a4ded60c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:28.975145 28393 consensus_queue.cc:260] T ba18c000cd3f42978ed8fd01a4ded60c P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:28.975797 28396 raft_consensus.cc:374] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:28.975978 28396 raft_consensus.cc:729] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:28.976591 28396 consensus_queue.cc:260] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:28.977095 28391 tablet_bootstrap.cc:654] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:28.977523 28390 tablet_bootstrap.cc:654] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:28.980610 28396 ts_tablet_manager.cc:1398] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.058s user 0.000s sys 0.003s
I20240502 01:54:28.980985 28396 tablet_bootstrap.cc:492] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:28.983525 28388 ts_tablet_manager.cc:1888] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:28.979336 28389 ts_tablet_manager.cc:1888] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:28.984539 28389 log.cc:1196] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/70e53c1d20c44e439eaa609615fa7137
I20240502 01:54:28.985116 28389 ts_tablet_manager.cc:1909] T 70e53c1d20c44e439eaa609615fa7137 P c2a28baa7ee2469294d94b7fe5455cff: Deleting consensus metadata
I20240502 01:54:28.984412 28394 tablet_bootstrap.cc:492] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:28.986999 28394 ts_tablet_manager.cc:1367] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.099s user 0.009s sys 0.000s
I20240502 01:54:28.987761 28394 raft_consensus.cc:348] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:28.988292 28394 raft_consensus.cc:374] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:28.988415 28394 raft_consensus.cc:729] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:28.988567 28396 tablet_bootstrap.cc:654] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:28.988754 28394 consensus_queue.cc:260] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:28.989385 28394 ts_tablet_manager.cc:1398] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:28.991636 28388 log.cc:1196] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/70e53c1d20c44e439eaa609615fa7137
I20240502 01:54:28.992471 28388 ts_tablet_manager.cc:1909] T 70e53c1d20c44e439eaa609615fa7137 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting consensus metadata
I20240502 01:54:28.994443 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 70e53c1d20c44e439eaa609615fa7137 (table rebalancer_dynamic_table_new_name [id=c9c501f8f45f4abca84bb4722d438787]) successfully deleted
I20240502 01:54:28.987731 28393 ts_tablet_manager.cc:1398] T ba18c000cd3f42978ed8fd01a4ded60c P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240502 01:54:28.995281 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet 70e53c1d20c44e439eaa609615fa7137 (table rebalancer_dynamic_table_new_name [id=c9c501f8f45f4abca84bb4722d438787]) successfully deleted
I20240502 01:54:28.998720 28391 tablet_bootstrap.cc:492] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:28.998991 28391 ts_tablet_manager.cc:1367] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.005s
I20240502 01:54:28.999385 28390 tablet_bootstrap.cc:492] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:28.999861 28391 raft_consensus.cc:348] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:29.000216 28390 ts_tablet_manager.cc:1367] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.001s
I20240502 01:54:29.000208 28391 raft_consensus.cc:374] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:29.000330 28391 raft_consensus.cc:729] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:29.000675 28391 consensus_queue.cc:260] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:29.004317 28391 ts_tablet_manager.cc:1398] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:54:29.004534 28390 raft_consensus.cc:348] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:29.004721 28390 raft_consensus.cc:374] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:29.004801 28390 raft_consensus.cc:729] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:29.005055 28390 consensus_queue.cc:260] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:29.006762 28390 ts_tablet_manager.cc:1398] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:54:29.007047 28390 tablet_bootstrap.cc:492] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:29.009796 28390 tablet_bootstrap.cc:654] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:29.017776 28390 tablet_bootstrap.cc:492] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:29.018026 28390 ts_tablet_manager.cc:1367] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20240502 01:54:29.018918 28390 raft_consensus.cc:348] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:29.019062 28395 tablet_bootstrap.cc:492] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:29.019196 28390 raft_consensus.cc:374] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:29.019261 28395 ts_tablet_manager.cc:1367] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.082s user 0.004s sys 0.000s
I20240502 01:54:29.019322 28390 raft_consensus.cc:729] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:29.019620 28390 consensus_queue.cc:260] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:29.020237 28395 raft_consensus.cc:348] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:29.020843 28395 raft_consensus.cc:374] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:29.021019 28395 raft_consensus.cc:729] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:29.021327 28395 consensus_queue.cc:260] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:29.022866 28395 ts_tablet_manager.cc:1398] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:29.027264 28390 ts_tablet_manager.cc:1398] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 01:54:29.032317 28396 tablet_bootstrap.cc:492] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:29.032500 28396 ts_tablet_manager.cc:1367] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.000s
I20240502 01:54:29.033354 28396 raft_consensus.cc:348] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:29.033581 28396 raft_consensus.cc:374] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:29.033710 28396 raft_consensus.cc:729] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:29.033969 28396 consensus_queue.cc:260] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:29.037729 28396 ts_tablet_manager.cc:1398] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:29.085829 27649 raft_consensus.cc:482] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:29.086040 27649 raft_consensus.cc:504] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:29.087136 27649 leader_election.cc:288] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:29.088102 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba18c000cd3f42978ed8fd01a4ded60c" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:29.088104 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba18c000cd3f42978ed8fd01a4ded60c" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:54:29.088413 24602 raft_consensus.cc:2454] T ba18c000cd3f42978ed8fd01a4ded60c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:54:29.088562 22118 raft_consensus.cc:2454] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:54:29.089617 22186 leader_election.cc:302] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:54:29.090405 27953 raft_consensus.cc:2781] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:29.090520 27953 raft_consensus.cc:482] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:29.090601 27953 raft_consensus.cc:3037] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:29.095822 27953 raft_consensus.cc:504] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:29.097046 27953 leader_election.cc:288] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:29.097728 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba18c000cd3f42978ed8fd01a4ded60c" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:54:29.098035 22118 raft_consensus.cc:3037] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:29.098023 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba18c000cd3f42978ed8fd01a4ded60c" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:29.098245 24603 raft_consensus.cc:3037] T ba18c000cd3f42978ed8fd01a4ded60c P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:29.100396 22118 raft_consensus.cc:2454] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:54:29.101140 24603 raft_consensus.cc:2454] T ba18c000cd3f42978ed8fd01a4ded60c P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:54:29.101632 22185 leader_election.cc:302] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:29.102190 27649 raft_consensus.cc:2781] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:29.102325 27649 raft_consensus.cc:686] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:54:29.102598 27649 consensus_queue.cc:237] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:29.117805 21707 catalog_manager.cc:5484] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:29.120810 22118 consensus_queue.cc:237] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:29.123917 21988 raft_consensus.cc:1261] T fedbec8860904a2cb5294037b7a4bcad P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.124109 25000 raft_consensus.cc:1261] T fedbec8860904a2cb5294037b7a4bcad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.125020 28229 consensus_queue.cc:1035] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.125747 28229 consensus_queue.cc:1035] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.128685 28234 raft_consensus.cc:2932] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:29.129307 25000 raft_consensus.cc:2932] T fedbec8860904a2cb5294037b7a4bcad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:29.130178 21988 raft_consensus.cc:2932] T fedbec8860904a2cb5294037b7a4bcad P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:29.138309 21715 catalog_manager.cc:5484] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:29.144830 21837 tablet_service.cc:1514] Processing DeleteTablet for tablet fedbec8860904a2cb5294037b7a4bcad with delete_type TABLET_DATA_TOMBSTONED (TS 759257ea7bef415ca698e60d82199d77 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:29.146215 28234 raft_consensus.cc:482] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:29.146526 28234 raft_consensus.cc:504] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:29.147558 28234 leader_election.cc:288] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:29.148075 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a82a89e92d243a78373077ea5a1f5de" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:29.148279 28435 tablet_replica.cc:330] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:29.148232 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a82a89e92d243a78373077ea5a1f5de" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:54:29.148309 24735 raft_consensus.cc:2454] T 5a82a89e92d243a78373077ea5a1f5de P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:54:29.148417 21988 raft_consensus.cc:2454] T 5a82a89e92d243a78373077ea5a1f5de P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:54:29.148499 28435 raft_consensus.cc:2229] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:29.148834 22053 leader_election.cc:302] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:54:29.148895 28435 raft_consensus.cc:2258] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:29.149160 28234 raft_consensus.cc:2781] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:29.149274 28234 raft_consensus.cc:482] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:29.149389 28234 raft_consensus.cc:3037] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:29.151669 28234 raft_consensus.cc:504] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:29.152194 28234 leader_election.cc:288] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:29.152704 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a82a89e92d243a78373077ea5a1f5de" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:54:29.152905 21988 raft_consensus.cc:3037] T 5a82a89e92d243a78373077ea5a1f5de P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:29.153002 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a82a89e92d243a78373077ea5a1f5de" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:29.153179 24735 raft_consensus.cc:3037] T 5a82a89e92d243a78373077ea5a1f5de P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:29.155215 28435 ts_tablet_manager.cc:1875] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:29.155632 21988 raft_consensus.cc:2454] T 5a82a89e92d243a78373077ea5a1f5de P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:54:29.156038 22053 leader_election.cc:302] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:54:29.156267 24735 raft_consensus.cc:2454] T 5a82a89e92d243a78373077ea5a1f5de P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:54:29.156348 28234 raft_consensus.cc:2781] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:29.156654 28234 raft_consensus.cc:686] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:54:29.157023 28234 consensus_queue.cc:237] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:29.161675 21711 catalog_manager.cc:5484] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:29.167135 28435 ts_tablet_manager.cc:1888] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:54:29.167358 28435 log.cc:1196] T fedbec8860904a2cb5294037b7a4bcad P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/fedbec8860904a2cb5294037b7a4bcad
I20240502 01:54:29.168064 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet fedbec8860904a2cb5294037b7a4bcad (table rebalance_test_table_13 [id=b462b431b51b4d45baf0ff5a495aa90a]) successfully deleted
I20240502 01:54:29.194701 28199 raft_consensus.cc:482] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:29.194890 28199 raft_consensus.cc:504] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:29.195528 28199 leader_election.cc:288] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:29.196209 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "641ed1f2bf7e40d7a8a6472cf9cd5519" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:29.196383 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "641ed1f2bf7e40d7a8a6472cf9cd5519" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:29.196455 25000 raft_consensus.cc:2454] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:29.196614 21856 raft_consensus.cc:2454] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:29.196849 24804 leader_election.cc:302] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:29.197261 28199 raft_consensus.cc:2781] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:29.197365 28199 raft_consensus.cc:482] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:29.197479 28199 raft_consensus.cc:3037] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:29.199975 28199 raft_consensus.cc:504] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:29.200552 28199 leader_election.cc:288] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:29.201037 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "641ed1f2bf7e40d7a8a6472cf9cd5519" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:29.201244 25000 raft_consensus.cc:3037] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:29.201351 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "641ed1f2bf7e40d7a8a6472cf9cd5519" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:29.201537 21857 raft_consensus.cc:3037] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:29.204053 25000 raft_consensus.cc:2454] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:29.204160 21857 raft_consensus.cc:2454] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:29.204530 24804 leader_election.cc:302] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:29.205581 28199 raft_consensus.cc:2781] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:29.205860 28199 raft_consensus.cc:686] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:29.206118 28199 consensus_queue.cc:237] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:29.210942 21711 catalog_manager.cc:5484] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:29.295010 28208 raft_consensus.cc:482] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:29.295217 28208 raft_consensus.cc:504] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:29.296239 28208 leader_election.cc:288] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:29.298074 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "791fbda0957e456082495e420d222d6b" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:54:29.298326 22118 raft_consensus.cc:2454] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:29.299094 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "791fbda0957e456082495e420d222d6b" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:29.299333 25000 raft_consensus.cc:2454] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:29.302639 24671 leader_election.cc:302] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:29.303037 28023 raft_consensus.cc:2781] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:29.303191 28023 raft_consensus.cc:482] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:29.303273 28023 raft_consensus.cc:3037] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:29.306303 28023 raft_consensus.cc:504] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:29.307475 28023 leader_election.cc:288] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:29.307690 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "791fbda0957e456082495e420d222d6b" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:29.308035 25000 raft_consensus.cc:3037] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:29.311069 25000 raft_consensus.cc:2454] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:29.311568 24671 leader_election.cc:302] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:29.311926 28023 raft_consensus.cc:2781] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:29.312065 28023 raft_consensus.cc:686] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:29.312381 28023 consensus_queue.cc:237] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:29.316692 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "791fbda0957e456082495e420d222d6b" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:54:29.316936 22120 raft_consensus.cc:3037] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:29.323441 22120 raft_consensus.cc:2454] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:29.338711 21707 catalog_manager.cc:5484] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:29.526726 21715 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:29.575265 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:54:29.582135 22120 raft_consensus.cc:1261] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:29.583428 25000 raft_consensus.cc:1261] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:29.584270 28023 consensus_queue.cc:1035] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:29.585148 28234 consensus_queue.cc:1035] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:29.590231 28023 consensus_queue.cc:1035] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:29.610328 28340 consensus_queue.cc:1035] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:29.650074 21712 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 791fbda0957e456082495e420d222d6b (table rebalancer_dynamic_table [id=8ba57952df5f43c5be003c4b1c675036]). Expected version 1 got 0
I20240502 01:54:29.671846 27649 consensus_queue.cc:1035] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:29.673985 28234 consensus_queue.cc:1035] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:29.694502 28422 consensus_queue.cc:1035] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:29.699609 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:29.725935 28511 tablet.cc:1722] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:29.731501 28510 tablet.cc:1722] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:29.733767 28510 tablet.cc:1620] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:29.734833 28511 tablet.cc:1620] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:29.735663 21712 catalog_manager.cc:5576] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879) does not have the latest schema for tablet 791fbda0957e456082495e420d222d6b (table rebalancer_dynamic_table [id=8ba57952df5f43c5be003c4b1c675036]). Expected version 1 got 0
I20240502 01:54:29.747126 21715 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:29.768321 24846 tablet_service.cc:1467] Processing CreateTablet for tablet ea103b58ddcc4459af3d51878a110d38 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b77487e3a60d4b7abb035e27c08af026]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:29.769474 21837 tablet_service.cc:1467] Processing CreateTablet for tablet ea103b58ddcc4459af3d51878a110d38 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b77487e3a60d4b7abb035e27c08af026]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:29.769917 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea103b58ddcc4459af3d51878a110d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:29.769902 21709 catalog_manager.cc:5576] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879) does not have the latest schema for tablet 791fbda0957e456082495e420d222d6b (table rebalancer_dynamic_table [id=8ba57952df5f43c5be003c4b1c675036]). Expected version 1 got 0
I20240502 01:54:29.768859 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea103b58ddcc4459af3d51878a110d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:29.773259 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 03fbd6c1568a44f38826a1c7ed9f7e99 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b77487e3a60d4b7abb035e27c08af026]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:29.773661 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03fbd6c1568a44f38826a1c7ed9f7e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:29.779031 22229 tablet_service.cc:1467] Processing CreateTablet for tablet 03fbd6c1568a44f38826a1c7ed9f7e99 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b77487e3a60d4b7abb035e27c08af026]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:29.779459 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03fbd6c1568a44f38826a1c7ed9f7e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:29.792989 21969 tablet_service.cc:1467] Processing CreateTablet for tablet 03fbd6c1568a44f38826a1c7ed9f7e99 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b77487e3a60d4b7abb035e27c08af026]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:29.793495 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03fbd6c1568a44f38826a1c7ed9f7e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:29.800948 24715 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:29.801012 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:29.805150 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 77a90f08d29a49d3b451f85cbdc828a6 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b77487e3a60d4b7abb035e27c08af026]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:29.805613 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a90f08d29a49d3b451f85cbdc828a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:29.814004 21709 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=8ba57952df5f43c5be003c4b1c675036] alter complete (version 1)
I20240502 01:54:29.816568 28199 consensus_queue.cc:1035] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:29.849895 24847 tablet_service.cc:1467] Processing CreateTablet for tablet 77a90f08d29a49d3b451f85cbdc828a6 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b77487e3a60d4b7abb035e27c08af026]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:29.850400 24847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a90f08d29a49d3b451f85cbdc828a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:29.863579 28528 tablet_bootstrap.cc:492] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:29.875741 28528 tablet_bootstrap.cc:654] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:29.891024 28528 tablet_bootstrap.cc:492] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:29.891362 28528 ts_tablet_manager.cc:1367] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.000s
I20240502 01:54:29.892419 28528 raft_consensus.cc:348] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:29.892802 28528 raft_consensus.cc:374] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:29.892987 28528 raft_consensus.cc:729] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:29.893373 28528 consensus_queue.cc:260] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:29.894218 28528 ts_tablet_manager.cc:1398] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:29.896344 28528 tablet_bootstrap.cc:492] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:29.900061 21837 tablet_service.cc:1467] Processing CreateTablet for tablet 77a90f08d29a49d3b451f85cbdc828a6 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b77487e3a60d4b7abb035e27c08af026]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:29.901281 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a90f08d29a49d3b451f85cbdc828a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:29.904507 28528 tablet_bootstrap.cc:654] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:29.905714 28536 tablet_bootstrap.cc:492] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:29.907274 28536 tablet_bootstrap.cc:654] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:29.922395 28528 tablet_bootstrap.cc:492] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:29.922744 28528 ts_tablet_manager.cc:1367] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.005s
I20240502 01:54:29.923866 28528 raft_consensus.cc:348] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:29.924391 28528 raft_consensus.cc:374] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:29.924551 28528 raft_consensus.cc:729] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:29.925019 28528 consensus_queue.cc:260] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:29.927443 28528 ts_tablet_manager.cc:1398] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:54:29.937911 28536 tablet_bootstrap.cc:492] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:29.938316 28536 ts_tablet_manager.cc:1367] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20240502 01:54:29.939464 28536 raft_consensus.cc:348] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:29.939854 28536 raft_consensus.cc:374] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:29.939947 28536 raft_consensus.cc:729] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:29.940268 28536 consensus_queue.cc:260] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:29.940632 28522 tablet.cc:1722] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:29.945701 28522 tablet.cc:1620] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a: MemRowSet was empty: no flush needed.
I20240502 01:54:29.948307 28536 ts_tablet_manager.cc:1398] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:54:29.966629 21709 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:29.980976 28510 tablet.cc:1722] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:29.981707 28510 tablet.cc:1620] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:29.982299 28527 tablet_bootstrap.cc:492] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:29.983536 28511 tablet.cc:1722] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:29.984226 28511 tablet.cc:1620] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:29.986755 28526 tablet_bootstrap.cc:492] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:29.988442 28526 tablet_bootstrap.cc:654] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:29.993170 28522 tablet.cc:1722] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:29.993904 28522 tablet.cc:1620] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a: MemRowSet was empty: no flush needed.
I20240502 01:54:29.997355 28533 tablet_bootstrap.cc:492] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:29.999183 28533 tablet_bootstrap.cc:654] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:30.030295 21709 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=8ba57952df5f43c5be003c4b1c675036] alter complete (version 2)
I20240502 01:54:30.058907 24583 tablet_service.cc:1467] Processing CreateTablet for tablet ea103b58ddcc4459af3d51878a110d38 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b77487e3a60d4b7abb035e27c08af026]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:30.059460 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea103b58ddcc4459af3d51878a110d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.078775 28527 tablet_bootstrap.cc:654] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:30.080813 22229 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:30.116161 28533 tablet_bootstrap.cc:492] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:30.116339 28533 ts_tablet_manager.cc:1367] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.136s user 0.003s sys 0.000s
I20240502 01:54:30.117106 28533 raft_consensus.cc:348] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:30.117328 28533 raft_consensus.cc:374] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:30.117383 28533 raft_consensus.cc:729] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:30.117596 28533 consensus_queue.cc:260] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:30.118114 28533 ts_tablet_manager.cc:1398] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:30.122670 28542 tablet_bootstrap.cc:492] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:30.124686 28542 tablet_bootstrap.cc:654] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
W20240502 01:54:30.126396 21923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.197:36677, user_credentials={real_user=slave}} blocked reactor thread for 83005.4us
I20240502 01:54:30.127722 28340 raft_consensus.cc:482] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:30.127971 28340 raft_consensus.cc:504] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:30.129144 28340 leader_election.cc:288] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:30.130476 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea103b58ddcc4459af3d51878a110d38" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:30.131184 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea103b58ddcc4459af3d51878a110d38" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
W20240502 01:54:30.131350 24802 leader_election.cc:341] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:54:30.131880 24801 leader_election.cc:341] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:30.132030 24801 leader_election.cc:302] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7449bb1df214bf4a5f93e4631312f08; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, 759257ea7bef415ca698e60d82199d77
I20240502 01:54:30.132346 28340 raft_consensus.cc:2726] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:30.136792 28527 tablet_bootstrap.cc:492] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:30.137135 28527 ts_tablet_manager.cc:1367] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.155s user 0.004s sys 0.000s
I20240502 01:54:30.137988 28527 raft_consensus.cc:348] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:30.138275 28527 raft_consensus.cc:374] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:30.138384 28527 raft_consensus.cc:729] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:30.138681 28527 consensus_queue.cc:260] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:30.139459 28527 ts_tablet_manager.cc:1398] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:30.149240 21709 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:30.151677 28526 tablet_bootstrap.cc:492] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:30.151899 28526 ts_tablet_manager.cc:1367] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.165s user 0.001s sys 0.003s
I20240502 01:54:30.153348 28526 raft_consensus.cc:348] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:30.153643 28526 raft_consensus.cc:374] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:30.153759 28526 raft_consensus.cc:729] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:30.154055 28526 consensus_queue.cc:260] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:30.154742 28526 ts_tablet_manager.cc:1398] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:30.155303 28526 tablet_bootstrap.cc:492] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:30.159329 28510 tablet.cc:1722] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:30.160733 28511 tablet.cc:1722] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:30.161149 21837 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:30.162401 28526 tablet_bootstrap.cc:654] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:30.171247 28522 tablet.cc:1722] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:30.181303 28340 raft_consensus.cc:482] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:30.181564 28340 raft_consensus.cc:504] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:30.182344 28340 leader_election.cc:288] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:30.183871 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a90f08d29a49d3b451f85cbdc828a6" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:30.184101 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a90f08d29a49d3b451f85cbdc828a6" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:54:30.184300 22118 raft_consensus.cc:2454] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 0.
I20240502 01:54:30.184762 24803 leader_election.cc:302] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:30.185649 28340 raft_consensus.cc:2781] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:30.185761 28340 raft_consensus.cc:482] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240502 01:54:30.185608 24801 leader_election.cc:341] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:30.185869 28340 raft_consensus.cc:3037] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:30.189045 21709 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=8ba57952df5f43c5be003c4b1c675036] alter complete (version 3)
I20240502 01:54:30.192525 28422 raft_consensus.cc:482] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:30.192799 28422 raft_consensus.cc:504] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:30.193377 28340 raft_consensus.cc:504] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:30.194411 28422 leader_election.cc:288] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:30.194777 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a90f08d29a49d3b451f85cbdc828a6" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:54:30.195001 22120 raft_consensus.cc:3037] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:30.200001 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fbd6c1568a44f38826a1c7ed9f7e99" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:54:30.200235 21988 raft_consensus.cc:2454] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:54:30.201226 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fbd6c1568a44f38826a1c7ed9f7e99" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:54:30.201462 21857 raft_consensus.cc:2454] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 0.
I20240502 01:54:30.202168 22184 leader_election.cc:302] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:30.202739 27649 raft_consensus.cc:2781] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:30.202840 27649 raft_consensus.cc:482] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:30.202937 27649 raft_consensus.cc:3037] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:30.209658 28542 tablet_bootstrap.cc:492] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:30.209832 28542 ts_tablet_manager.cc:1367] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.087s user 0.004s sys 0.000s
I20240502 01:54:30.211681 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:30.210742 28542 raft_consensus.cc:348] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:30.216154 27649 raft_consensus.cc:504] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:30.216459 28542 raft_consensus.cc:374] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:30.216599 28542 raft_consensus.cc:729] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:30.216969 28542 consensus_queue.cc:260] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:30.217121 27649 leader_election.cc:288] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:30.225692 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fbd6c1568a44f38826a1c7ed9f7e99" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:54:30.225914 21988 raft_consensus.cc:3037] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:30.228817 21988 raft_consensus.cc:2454] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:54:30.232666 22184 leader_election.cc:302] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:30.233335 27649 raft_consensus.cc:2781] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:30.233564 27649 raft_consensus.cc:686] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:54:30.233922 27649 consensus_queue.cc:237] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:30.235642 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a90f08d29a49d3b451f85cbdc828a6" candidate_uuid: "d7449bb1df214bf4a5f93e4631312f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:30.236420 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fbd6c1568a44f38826a1c7ed9f7e99" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:30.236608 21857 raft_consensus.cc:3037] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:54:30.236740 24801 leader_election.cc:341] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:30.237836 28340 leader_election.cc:288] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:30.243357 21857 raft_consensus.cc:2454] T 03fbd6c1568a44f38826a1c7ed9f7e99 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:54:30.244129 28542 ts_tablet_manager.cc:1398] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.034s user 0.003s sys 0.000s
I20240502 01:54:30.245860 28526 tablet_bootstrap.cc:492] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:30.246423 28526 ts_tablet_manager.cc:1367] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.091s user 0.003s sys 0.001s
I20240502 01:54:30.254230 28526 raft_consensus.cc:348] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:30.257108 22120 raft_consensus.cc:2454] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7449bb1df214bf4a5f93e4631312f08 in term 1.
I20240502 01:54:30.257834 28526 raft_consensus.cc:374] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:30.257987 28526 raft_consensus.cc:729] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:30.258313 28526 consensus_queue.cc:260] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:30.260506 28526 ts_tablet_manager.cc:1398] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 01:54:30.260833 28526 tablet_bootstrap.cc:492] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:30.273492 28526 tablet_bootstrap.cc:654] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:30.294018 21715 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:30.294370 21715 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:30.302395 28062 raft_consensus.cc:482] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:30.302578 28062 raft_consensus.cc:504] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:30.303494 28062 leader_election.cc:288] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:30.304242 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea103b58ddcc4459af3d51878a110d38" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:30.304502 24603 raft_consensus.cc:2454] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:54:30.305222 21791 leader_election.cc:302] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:30.306003 28062 raft_consensus.cc:2781] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:30.306083 28062 raft_consensus.cc:482] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:30.306142 28062 raft_consensus.cc:3037] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:30.306246 24865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea103b58ddcc4459af3d51878a110d38" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:30.306478 24865 raft_consensus.cc:2454] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 0.
I20240502 01:54:30.313112 28062 raft_consensus.cc:504] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:30.314002 28062 leader_election.cc:288] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:30.315593 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea103b58ddcc4459af3d51878a110d38" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:30.315930 24603 raft_consensus.cc:3037] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:30.320050 24603 raft_consensus.cc:2454] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:54:30.320683 21791 leader_election.cc:302] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:54:30.321552 24865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea103b58ddcc4459af3d51878a110d38" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:30.321796 24865 raft_consensus.cc:3037] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:30.322937 28062 raft_consensus.cc:2781] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:30.323164 28062 raft_consensus.cc:686] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:54:30.323531 28062 consensus_queue.cc:237] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:30.324906 21715 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 791fbda0957e456082495e420d222d6b
I20240502 01:54:30.325803 24715 tablet_service.cc:1514] Processing DeleteTablet for tablet 791fbda0957e456082495e420d222d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:30 UTC) from {username='slave'} at 127.0.0.1:41650
I20240502 01:54:30.326606 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 791fbda0957e456082495e420d222d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:30 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:30.329589 24865 raft_consensus.cc:2454] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:54:30.335795 24981 tablet_service.cc:1514] Processing DeleteTablet for tablet 791fbda0957e456082495e420d222d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:30 UTC) from {username='slave'} at 127.0.0.1:39964
I20240502 01:54:30.345844 24803 leader_election.cc:302] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a, d7449bb1df214bf4a5f93e4631312f08; no voters: 759257ea7bef415ca698e60d82199d77
I20240502 01:54:30.346316 28530 raft_consensus.cc:2781] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:30.346827 28530 raft_consensus.cc:686] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Becoming Leader. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Running, Role: LEADER
I20240502 01:54:30.347254 28530 consensus_queue.cc:237] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:30.350378 28555 tablet_replica.cc:330] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:54:30.350145 21715 catalog_manager.cc:5484] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:30.350612 28555 raft_consensus.cc:2229] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:30.351030 28555 raft_consensus.cc:2258] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:30.351573 28556 tablet_replica.cc:330] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:30.351779 28556 raft_consensus.cc:2229] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:30.351946 28556 raft_consensus.cc:2258] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:30.367180 28556 ts_tablet_manager.cc:1875] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:30.371714 28555 ts_tablet_manager.cc:1875] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:30.373915 28558 tablet_replica.cc:330] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:54:30.374229 28558 raft_consensus.cc:2229] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:30.374486 28558 raft_consensus.cc:2258] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:30.279202 21709 catalog_manager.cc:5484] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:30.375658 28558 ts_tablet_manager.cc:1875] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:30.387125 21715 catalog_manager.cc:5484] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199). New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:30.393667 28526 tablet_bootstrap.cc:492] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77: No bootstrap required, opened a new log
I20240502 01:54:30.394022 28526 ts_tablet_manager.cc:1367] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.133s user 0.003s sys 0.004s
I20240502 01:54:30.394981 28526 raft_consensus.cc:348] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:30.395391 24846 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:30.396908 28526 raft_consensus.cc:374] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:30.397040 28526 raft_consensus.cc:729] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:30.397372 28526 consensus_queue.cc:260] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:30.398773 28526 ts_tablet_manager.cc:1398] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:30.404071 28556 ts_tablet_manager.cc:1888] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:30.415385 28555 ts_tablet_manager.cc:1888] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:30.421172 28555 log.cc:1196] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/791fbda0957e456082495e420d222d6b
I20240502 01:54:30.427572 28555 ts_tablet_manager.cc:1909] T 791fbda0957e456082495e420d222d6b P 28157bc3e678428fb5f93c077d528513: Deleting consensus metadata
I20240502 01:54:30.429265 28558 ts_tablet_manager.cc:1888] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:30.430944 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet 791fbda0957e456082495e420d222d6b (table rebalancer_dynamic_table_new_name [id=8ba57952df5f43c5be003c4b1c675036]) successfully deleted
I20240502 01:54:30.431450 28556 log.cc:1196] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/791fbda0957e456082495e420d222d6b
I20240502 01:54:30.433557 28556 ts_tablet_manager.cc:1909] T 791fbda0957e456082495e420d222d6b P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:30.436841 21715 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:30.440168 28558 log.cc:1196] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/791fbda0957e456082495e420d222d6b
I20240502 01:54:30.440804 28558 ts_tablet_manager.cc:1909] T 791fbda0957e456082495e420d222d6b P 41c70f373c9d49a49891b03a12c6cb2b: Deleting consensus metadata
I20240502 01:54:30.444124 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 791fbda0957e456082495e420d222d6b (table rebalancer_dynamic_table_new_name [id=8ba57952df5f43c5be003c4b1c675036]) successfully deleted
I20240502 01:54:30.445667 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet 791fbda0957e456082495e420d222d6b (table rebalancer_dynamic_table_new_name [id=8ba57952df5f43c5be003c4b1c675036]) successfully deleted
I20240502 01:54:30.447263 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 897de384669a44df91d551d15a318627 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=518103d2e93246778fd5f5400e0bd8a5]), partition=
I20240502 01:54:30.447598 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897de384669a44df91d551d15a318627. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.447898 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 897de384669a44df91d551d15a318627 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=518103d2e93246778fd5f5400e0bd8a5]), partition=
I20240502 01:54:30.448278 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897de384669a44df91d551d15a318627. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.448236 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 897de384669a44df91d551d15a318627 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=518103d2e93246778fd5f5400e0bd8a5]), partition=
I20240502 01:54:30.448602 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897de384669a44df91d551d15a318627. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.453727 28536 tablet_bootstrap.cc:492] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:30.455137 28536 tablet_bootstrap.cc:654] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:30.456017 28542 tablet_bootstrap.cc:492] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:30.457723 28542 tablet_bootstrap.cc:654] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:30.474670 28542 tablet_bootstrap.cc:492] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f: No bootstrap required, opened a new log
I20240502 01:54:30.475009 28542 ts_tablet_manager.cc:1367] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.000s
I20240502 01:54:30.484084 28542 raft_consensus.cc:348] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:30.484498 28542 raft_consensus.cc:374] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:30.484634 28542 raft_consensus.cc:729] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:30.485016 28542 consensus_queue.cc:260] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:30.485637 28536 tablet_bootstrap.cc:492] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a: No bootstrap required, opened a new log
I20240502 01:54:30.485909 28536 ts_tablet_manager.cc:1367] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.000s
I20240502 01:54:30.486757 28536 raft_consensus.cc:348] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:30.487121 28536 raft_consensus.cc:374] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:30.487272 28536 raft_consensus.cc:729] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:30.487577 28536 consensus_queue.cc:260] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:30.491653 28561 tablet_bootstrap.cc:492] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:30.492576 28229 raft_consensus.cc:482] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:30.492707 28229 raft_consensus.cc:504] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:30.493379 28229 leader_election.cc:288] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:30.493542 28542 ts_tablet_manager.cc:1398] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240502 01:54:30.494449 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897de384669a44df91d551d15a318627" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:30.494701 24603 raft_consensus.cc:2454] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 0.
I20240502 01:54:30.495214 22054 leader_election.cc:302] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:54:30.495723 28229 raft_consensus.cc:2781] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:30.495805 28229 raft_consensus.cc:482] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:30.495874 28229 raft_consensus.cc:3037] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:30.498648 28229 raft_consensus.cc:504] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:30.500475 28536 ts_tablet_manager.cc:1398] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:30.502974 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897de384669a44df91d551d15a318627" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:30.503180 24603 raft_consensus.cc:3037] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:30.506356 24603 raft_consensus.cc:2454] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:54:30.506873 22054 leader_election.cc:302] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:54:30.507138 28234 raft_consensus.cc:2781] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:30.509111 28229 leader_election.cc:288] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:30.494153 24734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897de384669a44df91d551d15a318627" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:54:30.499678 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897de384669a44df91d551d15a318627" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
W20240502 01:54:30.513036 22054 leader_election.cc:341] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:54:30.513440 22054 leader_election.cc:341] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:30.521695 28234 raft_consensus.cc:686] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:54:30.521989 28234 consensus_queue.cc:237] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:30.526101 28561 tablet_bootstrap.cc:654] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:30.529911 28062 raft_consensus.cc:482] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:30.530043 28062 raft_consensus.cc:504] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:30.530658 28062 leader_election.cc:288] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:30.530973 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a90f08d29a49d3b451f85cbdc828a6" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:54:30.531275 22120 raft_consensus.cc:2379] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 759257ea7bef415ca698e60d82199d77 in current term 1: Already voted for candidate d7449bb1df214bf4a5f93e4631312f08 in this term.
I20240502 01:54:30.531921 24866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a90f08d29a49d3b451f85cbdc828a6" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:30.532524 21791 leader_election.cc:302] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: c64840657cdf4b93980bd34120c94d9a, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:30.532934 28062 raft_consensus.cc:3037] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:30.535677 28062 raft_consensus.cc:2726] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:30.535566 21715 catalog_manager.cc:5484] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 0 to 1, leader changed from <none> to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:30.558641 28561 tablet_bootstrap.cc:492] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:30.558900 28561 ts_tablet_manager.cc:1367] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.000s
I20240502 01:54:30.560583 28561 raft_consensus.cc:348] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:30.561084 28561 raft_consensus.cc:374] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:30.561264 28561 raft_consensus.cc:729] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:30.561555 28561 consensus_queue.cc:260] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:30.562652 28561 ts_tablet_manager.cc:1398] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:30.615594 28502 raft_consensus.cc:482] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:30.615797 28502 raft_consensus.cc:504] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:30.616681 28502 leader_election.cc:288] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:54:30.617565 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897de384669a44df91d551d15a318627" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:54:30.617625 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897de384669a44df91d551d15a318627" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:54:30.617967 24603 raft_consensus.cc:2379] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28157bc3e678428fb5f93c077d528513 in current term 1: Already voted for candidate c64840657cdf4b93980bd34120c94d9a in this term.
I20240502 01:54:30.619582 24669 leader_election.cc:302] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28157bc3e678428fb5f93c077d528513; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, c64840657cdf4b93980bd34120c94d9a
I20240502 01:54:30.620827 28546 raft_consensus.cc:3037] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:30.623785 28546 raft_consensus.cc:2726] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:30.638877 22120 consensus_queue.cc:237] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:30.641041 24732 raft_consensus.cc:1261] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:30.641189 22251 raft_consensus.cc:1261] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:30.641667 28564 consensus_queue.cc:1035] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:30.641942 28234 consensus_queue.cc:1035] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:30.644810 28234 raft_consensus.cc:2932] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:30.645617 22251 raft_consensus.cc:2932] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:30.645727 24732 raft_consensus.cc:2932] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:30.651800 21715 catalog_manager.cc:5484] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:30.655218 21712 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:30.658488 21969 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eed9ee06a494588a32b4c951d6d3a14 with delete_type TABLET_DATA_TOMBSTONED (TS 06543b05a16f42099bc86db2a0284412 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:30.659209 28588 tablet_replica.cc:330] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:30.659456 28588 raft_consensus.cc:2229] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:30.659754 28588 raft_consensus.cc:2258] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:30.660791 28588 ts_tablet_manager.cc:1875] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:30.662192 24603 raft_consensus.cc:1261] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:30.662846 28229 consensus_queue.cc:1035] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:30.665326 24732 raft_consensus.cc:1261] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:30.666049 28234 consensus_queue.cc:1035] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:30.673604 28588 ts_tablet_manager.cc:1888] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:54:30.677829 28588 log.cc:1196] T 0eed9ee06a494588a32b4c951d6d3a14 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/0eed9ee06a494588a32b4c951d6d3a14
I20240502 01:54:30.681949 21712 catalog_manager.cc:5576] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677) does not have the latest schema for tablet 897de384669a44df91d551d15a318627 (table rebalancer_dynamic_table [id=518103d2e93246778fd5f5400e0bd8a5]). Expected version 1 got 0
I20240502 01:54:30.682782 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 0eed9ee06a494588a32b4c951d6d3a14 (table rebalance_test_table_1 [id=eaf29e0201fc4b738b1b30c13a773ffd]) successfully deleted
I20240502 01:54:30.683660 28522 tablet.cc:1722] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:30.684139 28522 tablet.cc:1620] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a: MemRowSet was empty: no flush needed.
I20240502 01:54:30.685849 28510 tablet.cc:1722] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:30.686358 28510 tablet.cc:1620] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:30.688241 21712 catalog_manager.cc:5576] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375) does not have the latest schema for tablet 897de384669a44df91d551d15a318627 (table rebalancer_dynamic_table [id=518103d2e93246778fd5f5400e0bd8a5]). Expected version 1 got 0
I20240502 01:54:30.695149 21712 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=518103d2e93246778fd5f5400e0bd8a5] alter complete (version 1)
I20240502 01:54:30.702934 21711 catalog_manager.cc:5576] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677) does not have the latest schema for tablet 897de384669a44df91d551d15a318627 (table rebalancer_dynamic_table [id=518103d2e93246778fd5f5400e0bd8a5]). Expected version 1 got 0
I20240502 01:54:30.707927 28595 tablet.cc:1722] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:30.708664 28595 tablet.cc:1620] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:30.746213 28517 consensus_queue.cc:1035] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:30.767465 27649 consensus_queue.cc:1035] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:30.782562 28560 consensus_queue.cc:1035] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:30.789425 28489 consensus_queue.cc:1035] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:30.802716 21711 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:30.816073 28522 tablet.cc:1722] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:30.816673 28522 tablet.cc:1620] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a: MemRowSet was empty: no flush needed.
I20240502 01:54:30.819151 28595 tablet.cc:1722] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:30.819649 28595 tablet.cc:1620] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f: MemRowSet was empty: no flush needed.
I20240502 01:54:30.823925 28510 tablet.cc:1722] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:30.824383 28510 tablet.cc:1620] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513: MemRowSet was empty: no flush needed.
I20240502 01:54:30.830268 21712 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=518103d2e93246778fd5f5400e0bd8a5] alter complete (version 2)
I20240502 01:54:30.863696 28340 consensus_queue.cc:1035] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:30.888973 28530 consensus_queue.cc:1035] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:30.910426 21712 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:30.936105 22229 tablet_service.cc:1467] Processing CreateTablet for tablet 9b186753ceeb4a3c9f22391bb9a5b25f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=aafacdd1649640daab7dbb1308258509]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:30.936177 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 9b186753ceeb4a3c9f22391bb9a5b25f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=aafacdd1649640daab7dbb1308258509]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:30.936362 24846 tablet_service.cc:1467] Processing CreateTablet for tablet 9b186753ceeb4a3c9f22391bb9a5b25f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=aafacdd1649640daab7dbb1308258509]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:30.936725 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b186753ceeb4a3c9f22391bb9a5b25f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.936724 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b186753ceeb4a3c9f22391bb9a5b25f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.936749 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b186753ceeb4a3c9f22391bb9a5b25f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.943250 28561 tablet_bootstrap.cc:492] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:30.944950 28561 tablet_bootstrap.cc:654] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:30.945986 28625 tablet_bootstrap.cc:492] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:30.947790 28625 tablet_bootstrap.cc:654] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:30.948545 21715 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:30.952349 24847 tablet_service.cc:1467] Processing CreateTablet for tablet 45ccb387e833453a950b1ef5b5282d29 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=aafacdd1649640daab7dbb1308258509]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:30.952505 22229 tablet_service.cc:1467] Processing CreateTablet for tablet 45ccb387e833453a950b1ef5b5282d29 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=aafacdd1649640daab7dbb1308258509]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:30.952821 24847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ccb387e833453a950b1ef5b5282d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.953150 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ccb387e833453a950b1ef5b5282d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.955708 21969 tablet_service.cc:1467] Processing CreateTablet for tablet fa8220d5cb8140e6a66a3b377df56fe4 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=aafacdd1649640daab7dbb1308258509]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:30.956163 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8220d5cb8140e6a66a3b377df56fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.964397 24981 tablet_service.cc:1467] Processing CreateTablet for tablet fa8220d5cb8140e6a66a3b377df56fe4 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=aafacdd1649640daab7dbb1308258509]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:30.964978 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8220d5cb8140e6a66a3b377df56fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.965441 28625 tablet_bootstrap.cc:492] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:30.965572 28625 ts_tablet_manager.cc:1367] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240502 01:54:30.966284 28625 raft_consensus.cc:348] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:30.966655 28625 raft_consensus.cc:374] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:30.966708 28625 raft_consensus.cc:729] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:30.966950 28625 consensus_queue.cc:260] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:30.968621 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 45ccb387e833453a950b1ef5b5282d29 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=aafacdd1649640daab7dbb1308258509]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:30.969203 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ccb387e833453a950b1ef5b5282d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.969336 28625 ts_tablet_manager.cc:1398] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:54:30.969573 28625 tablet_bootstrap.cc:492] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:30.972239 28625 tablet_bootstrap.cc:654] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:30.973767 28522 tablet.cc:1722] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:30.977181 28595 tablet.cc:1722] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:30.986128 28628 tablet_bootstrap.cc:492] T 45ccb387e833453a950b1ef5b5282d29 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:30.988349 28628 tablet_bootstrap.cc:654] T 45ccb387e833453a950b1ef5b5282d29 P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:30.990267 28510 tablet.cc:1722] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:30.996223 24715 tablet_service.cc:1467] Processing CreateTablet for tablet fa8220d5cb8140e6a66a3b377df56fe4 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=aafacdd1649640daab7dbb1308258509]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:31.001585 21709 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=518103d2e93246778fd5f5400e0bd8a5] alter complete (version 3)
I20240502 01:54:31.005034 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8220d5cb8140e6a66a3b377df56fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:31.015074 28633 tablet_bootstrap.cc:492] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:31.019559 28633 tablet_bootstrap.cc:654] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:31.026007 28628 tablet_bootstrap.cc:492] T 45ccb387e833453a950b1ef5b5282d29 P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:31.026348 28628 ts_tablet_manager.cc:1367] T 45ccb387e833453a950b1ef5b5282d29 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.000s
I20240502 01:54:31.028983 28561 tablet_bootstrap.cc:492] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:31.029181 28561 ts_tablet_manager.cc:1367] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.086s user 0.002s sys 0.011s
I20240502 01:54:31.029911 28628 raft_consensus.cc:348] T 45ccb387e833453a950b1ef5b5282d29 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:31.029956 28561 raft_consensus.cc:348] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:31.030139 28628 raft_consensus.cc:374] T 45ccb387e833453a950b1ef5b5282d29 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:31.030174 28561 raft_consensus.cc:374] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:31.030238 28628 raft_consensus.cc:729] T 45ccb387e833453a950b1ef5b5282d29 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:31.030251 28561 raft_consensus.cc:729] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:31.030489 28561 consensus_queue.cc:260] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:31.031042 28628 consensus_queue.cc:260] T 45ccb387e833453a950b1ef5b5282d29 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:31.031921 28628 ts_tablet_manager.cc:1398] T 45ccb387e833453a950b1ef5b5282d29 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:31.032280 28628 tablet_bootstrap.cc:492] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:31.033442 28561 ts_tablet_manager.cc:1398] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:31.033828 28561 tablet_bootstrap.cc:492] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:31.036051 28628 tablet_bootstrap.cc:654] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:31.043953 28633 tablet_bootstrap.cc:492] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:31.044229 28633 ts_tablet_manager.cc:1367] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.004s
I20240502 01:54:31.047489 28633 raft_consensus.cc:348] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:31.047879 28633 raft_consensus.cc:374] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:31.048040 28633 raft_consensus.cc:729] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:31.048357 28633 consensus_queue.cc:260] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:31.049229 28633 ts_tablet_manager.cc:1398] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:54:31.057734 28628 tablet_bootstrap.cc:492] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08: No bootstrap required, opened a new log
I20240502 01:54:31.057986 28628 ts_tablet_manager.cc:1367] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.011s
I20240502 01:54:31.058925 28628 raft_consensus.cc:348] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:31.059250 28628 raft_consensus.cc:374] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:31.059499 28628 raft_consensus.cc:729] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:31.059547 28561 tablet_bootstrap.cc:654] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:31.059895 28628 consensus_queue.cc:260] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:31.060663 28628 ts_tablet_manager.cc:1398] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:31.076262 28630 tablet_bootstrap.cc:492] T fa8220d5cb8140e6a66a3b377df56fe4 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:31.078866 28630 tablet_bootstrap.cc:654] T fa8220d5cb8140e6a66a3b377df56fe4 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:31.080132 28561 tablet_bootstrap.cc:492] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:31.080298 28561 ts_tablet_manager.cc:1367] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.002s
I20240502 01:54:31.081022 28561 raft_consensus.cc:348] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:31.087945 28502 raft_consensus.cc:482] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:31.088315 28502 raft_consensus.cc:504] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:31.088973 28502 leader_election.cc:288] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:31.089576 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b186753ceeb4a3c9f22391bb9a5b25f" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:31.089823 22251 raft_consensus.cc:2454] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:31.091888 24670 leader_election.cc:302] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:31.093497 24865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b186753ceeb4a3c9f22391bb9a5b25f" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:31.093753 24865 raft_consensus.cc:2454] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:31.098627 28502 raft_consensus.cc:2781] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:31.098743 28502 raft_consensus.cc:482] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:31.098850 28502 raft_consensus.cc:3037] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:31.100993 28561 raft_consensus.cc:374] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:31.101127 28561 raft_consensus.cc:729] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:31.101382 28561 consensus_queue.cc:260] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:31.101792 28502 raft_consensus.cc:504] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:31.102262 28561 ts_tablet_manager.cc:1398] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240502 01:54:31.103257 24865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b186753ceeb4a3c9f22391bb9a5b25f" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:31.103473 24865 raft_consensus.cc:3037] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:31.104413 28561 tablet_bootstrap.cc:492] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:31.105038 21712 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:31.105243 21712 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:31.106608 28561 tablet_bootstrap.cc:654] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:31.107766 28625 tablet_bootstrap.cc:492] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:31.107928 28625 ts_tablet_manager.cc:1367] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.138s user 0.024s sys 0.004s
I20240502 01:54:31.109143 21712 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: Sending DeleteTablet for 3 replicas of tablet 897de384669a44df91d551d15a318627
I20240502 01:54:31.108985 28625 raft_consensus.cc:348] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:31.109321 28625 raft_consensus.cc:374] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:31.109371 28625 raft_consensus.cc:729] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:31.109935 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 897de384669a44df91d551d15a318627 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:31 UTC) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:31.110608 24583 tablet_service.cc:1514] Processing DeleteTablet for tablet 897de384669a44df91d551d15a318627 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:31 UTC) from {username='slave'} at 127.0.0.1:53832
I20240502 01:54:31.111846 24715 tablet_service.cc:1514] Processing DeleteTablet for tablet 897de384669a44df91d551d15a318627 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:31 UTC) from {username='slave'} at 127.0.0.1:41650
I20240502 01:54:31.113036 28625 consensus_queue.cc:260] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:31.114024 28625 ts_tablet_manager.cc:1398] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:54:31.116294 24865 raft_consensus.cc:2454] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:31.119755 28502 leader_election.cc:288] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:31.122835 28646 tablet_replica.cc:330] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:54:31.123067 28646 raft_consensus.cc:2229] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:31.123304 28646 raft_consensus.cc:2258] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:31.123903 28647 tablet_replica.cc:330] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:54:31.124297 28647 raft_consensus.cc:2229] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:31.124573 28647 raft_consensus.cc:2258] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:31.125334 28646 ts_tablet_manager.cc:1875] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:31.126325 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b186753ceeb4a3c9f22391bb9a5b25f" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:31.126514 22251 raft_consensus.cc:3037] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:31.131664 28646 ts_tablet_manager.cc:1888] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:31.131801 28646 log.cc:1196] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/897de384669a44df91d551d15a318627
I20240502 01:54:31.132294 28646 ts_tablet_manager.cc:1909] T 897de384669a44df91d551d15a318627 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting consensus metadata
I20240502 01:54:31.133828 28647 ts_tablet_manager.cc:1875] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:31.140218 28647 ts_tablet_manager.cc:1888] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:31.140408 28647 log.cc:1196] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/897de384669a44df91d551d15a318627
I20240502 01:54:31.141965 28647 ts_tablet_manager.cc:1909] T 897de384669a44df91d551d15a318627 P 28157bc3e678428fb5f93c077d528513: Deleting consensus metadata
I20240502 01:54:31.142976 24669 leader_election.cc:302] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:31.143275 28502 raft_consensus.cc:2781] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:31.143440 28502 raft_consensus.cc:686] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:31.143743 28502 consensus_queue.cc:237] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:31.144027 28630 tablet_bootstrap.cc:492] T fa8220d5cb8140e6a66a3b377df56fe4 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:31.144197 28561 tablet_bootstrap.cc:492] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513: No bootstrap required, opened a new log
I20240502 01:54:31.144184 28630 ts_tablet_manager.cc:1367] T fa8220d5cb8140e6a66a3b377df56fe4 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.003s
I20240502 01:54:31.144464 28561 ts_tablet_manager.cc:1367] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.040s user 0.014s sys 0.010s
I20240502 01:54:31.145678 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet 897de384669a44df91d551d15a318627 (table rebalancer_dynamic_table_new_name [id=518103d2e93246778fd5f5400e0bd8a5]) successfully deleted
I20240502 01:54:31.145977 28561 raft_consensus.cc:348] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:31.146325 28561 raft_consensus.cc:374] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:31.146481 28561 raft_consensus.cc:729] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:31.146847 28561 consensus_queue.cc:260] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:31.147141 28645 tablet_replica.cc:330] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:31.147640 28561 ts_tablet_manager.cc:1398] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:31.147555 28630 raft_consensus.cc:348] T fa8220d5cb8140e6a66a3b377df56fe4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:31.147446 28645 raft_consensus.cc:2229] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:31.148204 28645 raft_consensus.cc:2258] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:31.148682 28630 raft_consensus.cc:374] T fa8220d5cb8140e6a66a3b377df56fe4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:31.148831 28630 raft_consensus.cc:729] T fa8220d5cb8140e6a66a3b377df56fe4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:31.149154 28630 consensus_queue.cc:260] T fa8220d5cb8140e6a66a3b377df56fe4 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:31.149572 28630 ts_tablet_manager.cc:1398] T fa8220d5cb8140e6a66a3b377df56fe4 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:54:31.150285 28645 ts_tablet_manager.cc:1875] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:31.151787 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet 897de384669a44df91d551d15a318627 (table rebalancer_dynamic_table_new_name [id=518103d2e93246778fd5f5400e0bd8a5]) successfully deleted
I20240502 01:54:31.155745 22251 raft_consensus.cc:2454] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:31.159922 28645 ts_tablet_manager.cc:1888] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:31.161495 28645 log.cc:1196] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/897de384669a44df91d551d15a318627
I20240502 01:54:31.161963 28645 ts_tablet_manager.cc:1909] T 897de384669a44df91d551d15a318627 P c64840657cdf4b93980bd34120c94d9a: Deleting consensus metadata
I20240502 01:54:31.164175 21709 catalog_manager.cc:5484] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:31.165199 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 897de384669a44df91d551d15a318627 (table rebalancer_dynamic_table_new_name [id=518103d2e93246778fd5f5400e0bd8a5]) successfully deleted
I20240502 01:54:31.212134 21709 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33848:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:31.221290 24981 tablet_service.cc:1467] Processing CreateTablet for tablet 69f7f6d27ad1486393b60321f8efefb3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=535c7a299048459fb4fc788132d58bd7]), partition=
I20240502 01:54:31.221469 22229 tablet_service.cc:1467] Processing CreateTablet for tablet 69f7f6d27ad1486393b60321f8efefb3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=535c7a299048459fb4fc788132d58bd7]), partition=
I20240502 01:54:31.221724 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69f7f6d27ad1486393b60321f8efefb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:31.221915 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69f7f6d27ad1486393b60321f8efefb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:31.228682 21969 tablet_service.cc:1467] Processing CreateTablet for tablet 69f7f6d27ad1486393b60321f8efefb3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=535c7a299048459fb4fc788132d58bd7]), partition=
I20240502 01:54:31.229107 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69f7f6d27ad1486393b60321f8efefb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:31.231344 28625 tablet_bootstrap.cc:492] T 69f7f6d27ad1486393b60321f8efefb3 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:31.232477 28625 tablet_bootstrap.cc:654] T 69f7f6d27ad1486393b60321f8efefb3 P c2a28baa7ee2469294d94b7fe5455cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:31.233110 28502 raft_consensus.cc:482] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:31.233282 28502 raft_consensus.cc:504] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:31.234170 28502 leader_election.cc:288] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:31.234869 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8220d5cb8140e6a66a3b377df56fe4" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:31.235203 25001 raft_consensus.cc:2454] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:31.235780 24671 leader_election.cc:302] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:31.236410 28502 raft_consensus.cc:2781] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:31.236536 28502 raft_consensus.cc:482] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:31.236624 28502 raft_consensus.cc:3037] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:31.239339 21987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8220d5cb8140e6a66a3b377df56fe4" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:54:31.239573 21987 raft_consensus.cc:2454] T fa8220d5cb8140e6a66a3b377df56fe4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:31.241165 28502 raft_consensus.cc:504] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:31.241950 28502 leader_election.cc:288] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:31.243345 28630 tablet_bootstrap.cc:492] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:31.244801 28630 tablet_bootstrap.cc:654] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:31.246495 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8220d5cb8140e6a66a3b377df56fe4" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:31.246754 25001 raft_consensus.cc:3037] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:31.249527 25001 raft_consensus.cc:2454] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:31.250968 24671 leader_election.cc:302] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:31.251713 28502 raft_consensus.cc:2781] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:31.252005 28502 raft_consensus.cc:686] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:31.252363 28502 consensus_queue.cc:237] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:31.255189 28630 tablet_bootstrap.cc:492] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412: No bootstrap required, opened a new log
I20240502 01:54:31.255445 28630 ts_tablet_manager.cc:1367] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20240502 01:54:31.256294 28630 raft_consensus.cc:348] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:31.256701 28630 raft_consensus.cc:374] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:31.256845 28630 raft_consensus.cc:729] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:31.257138 28630 consensus_queue.cc:260] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:31.257910 28630 ts_tablet_manager.cc:1398] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:31.260825 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8220d5cb8140e6a66a3b377df56fe4" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:54:31.261040 21988 raft_consensus.cc:3037] T fa8220d5cb8140e6a66a3b377df56fe4 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:31.263259 21988 raft_consensus.cc:2454] T fa8220d5cb8140e6a66a3b377df56fe4 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:31.272634 28633 tablet_bootstrap.cc:492] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:31.274350 28633 tablet_bootstrap.cc:654] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:31.284390 28633 tablet_bootstrap.cc:492] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b: No bootstrap required, opened a new log
I20240502 01:54:31.284613 28633 ts_tablet_manager.cc:1367] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 01:54:31.285532 28633 raft_consensus.cc:348] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:31.285773 28633 raft_consensus.cc:374] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:31.286134 28633 raft_consensus.cc:729] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:31.286592 28633 consensus_queue.cc:260] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:31.287744 28633 ts_tablet_manager.cc:1398] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:54:31.305663 21708 catalog_manager.cc:5484] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:31.306620 28625 tablet_bootstrap.cc:492] T 69f7f6d27ad1486393b60321f8efefb3 P c2a28baa7ee2469294d94b7fe5455cff: No bootstrap required, opened a new log
I20240502 01:54:31.306764 28625 ts_tablet_manager.cc:1367] T 69f7f6d27ad1486393b60321f8efefb3 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.075s user 0.009s sys 0.004s
I20240502 01:54:31.307441 28625 raft_consensus.cc:348] T 69f7f6d27ad1486393b60321f8efefb3 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:31.307710 28625 raft_consensus.cc:374] T 69f7f6d27ad1486393b60321f8efefb3 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:31.307775 28625 raft_consensus.cc:729] T 69f7f6d27ad1486393b60321f8efefb3 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:31.308007 28625 consensus_queue.cc:260] T 69f7f6d27ad1486393b60321f8efefb3 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:31.310817 28625 ts_tablet_manager.cc:1398] T 69f7f6d27ad1486393b60321f8efefb3 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:54:31.324323 28540 raft_consensus.cc:482] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:31.324497 28540 raft_consensus.cc:504] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:31.325297 28540 leader_election.cc:288] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:31.326228 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f7f6d27ad1486393b60321f8efefb3" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:54:31.326472 25001 raft_consensus.cc:2454] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:31.327013 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f7f6d27ad1486393b60321f8efefb3" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:31.327219 22251 raft_consensus.cc:2454] T 69f7f6d27ad1486393b60321f8efefb3 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 0.
I20240502 01:54:31.327715 21925 leader_election.cc:302] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:54:31.328435 28669 raft_consensus.cc:2781] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:31.328552 28669 raft_consensus.cc:482] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:31.328665 28669 raft_consensus.cc:3037] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:31.331300 28669 raft_consensus.cc:504] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:31.332338 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f7f6d27ad1486393b60321f8efefb3" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:31.332547 25001 raft_consensus.cc:3037] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:31.333274 28669 leader_election.cc:288] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:31.333320 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f7f6d27ad1486393b60321f8efefb3" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:31.333506 22251 raft_consensus.cc:3037] T 69f7f6d27ad1486393b60321f8efefb3 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:31.336206 22251 raft_consensus.cc:2454] T 69f7f6d27ad1486393b60321f8efefb3 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:31.336748 21924 leader_election.cc:302] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:31.344977 28669 raft_consensus.cc:2781] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:31.345165 28669 raft_consensus.cc:686] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:54:31.345436 28669 consensus_queue.cc:237] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:31.347282 25001 raft_consensus.cc:2454] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
I20240502 01:54:31.359541 21715 catalog_manager.cc:5484] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:31.423449 28546 raft_consensus.cc:482] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:31.423641 28546 raft_consensus.cc:504] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:31.424391 28546 leader_election.cc:288] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:31.425412 24865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ccb387e833453a950b1ef5b5282d29" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08" is_pre_election: true
I20240502 01:54:31.425679 24865 raft_consensus.cc:2454] T 45ccb387e833453a950b1ef5b5282d29 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:31.427320 24669 leader_election.cc:302] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:31.427667 28546 raft_consensus.cc:2781] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:31.427717 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ccb387e833453a950b1ef5b5282d29" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:54:31.427809 28546 raft_consensus.cc:482] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:31.427922 28546 raft_consensus.cc:3037] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:31.427949 22249 raft_consensus.cc:2454] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 0.
I20240502 01:54:31.430965 28546 raft_consensus.cc:504] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:31.432344 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ccb387e833453a950b1ef5b5282d29" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:54:31.432545 22249 raft_consensus.cc:3037] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:31.432569 24865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ccb387e833453a950b1ef5b5282d29" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:31.432822 24865 raft_consensus.cc:3037] T 45ccb387e833453a950b1ef5b5282d29 P d7449bb1df214bf4a5f93e4631312f08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:31.435199 22249 raft_consensus.cc:2454] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:31.435873 24865 raft_consensus.cc:2454] T 45ccb387e833453a950b1ef5b5282d29 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:54:31.436427 24670 leader_election.cc:302] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:54:31.437343 28502 raft_consensus.cc:2781] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:31.437623 28502 raft_consensus.cc:686] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:31.437999 28502 consensus_queue.cc:237] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:31.460718 28546 leader_election.cc:288] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 1 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:31.468986 21715 catalog_manager.cc:5484] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:31.479001 21715 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:31.503885 22249 raft_consensus.cc:1261] T 69f7f6d27ad1486393b60321f8efefb3 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:31.504500 25001 raft_consensus.cc:1261] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:31.504743 28669 consensus_queue.cc:1035] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:31.505945 28669 consensus_queue.cc:1035] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:31.517865 21708 catalog_manager.cc:5576] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093) does not have the latest schema for tablet 69f7f6d27ad1486393b60321f8efefb3 (table rebalancer_dynamic_table [id=535c7a299048459fb4fc788132d58bd7]). Expected version 1 got 0
I20240502 01:54:31.526623 28701 tablet.cc:1722] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:31.536012 28685 consensus_queue.cc:1035] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:31.532423 28701 tablet.cc:1620] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412: MemRowSet was empty: no flush needed.
I20240502 01:54:31.538272 28702 tablet.cc:1722] T 69f7f6d27ad1486393b60321f8efefb3 P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:31.544346 21712 catalog_manager.cc:5576] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093) does not have the latest schema for tablet 69f7f6d27ad1486393b60321f8efefb3 (table rebalancer_dynamic_table [id=535c7a299048459fb4fc788132d58bd7]). Expected version 1 got 0
I20240502 01:54:31.551868 28702 tablet.cc:1620] T 69f7f6d27ad1486393b60321f8efefb3 P c2a28baa7ee2469294d94b7fe5455cff: MemRowSet was empty: no flush needed.
I20240502 01:54:31.558197 21712 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 69f7f6d27ad1486393b60321f8efefb3 (table rebalancer_dynamic_table [id=535c7a299048459fb4fc788132d58bd7]). Expected version 1 got 0
I20240502 01:54:31.565822 28705 tablet.cc:1722] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:31.570514 21712 catalog_manager.cc:5576] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097) does not have the latest schema for tablet 69f7f6d27ad1486393b60321f8efefb3 (table rebalancer_dynamic_table [id=535c7a299048459fb4fc788132d58bd7]). Expected version 1 got 0
I20240502 01:54:31.570034 28705 tablet.cc:1620] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:31.585284 21712 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=535c7a299048459fb4fc788132d58bd7] alter complete (version 1)
I20240502 01:54:31.628290 28685 consensus_queue.cc:1035] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:31.705832 21715 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:31.721423 28701 tablet.cc:1722] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:31.721485 28705 tablet.cc:1722] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:31.721951 28701 tablet.cc:1620] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412: MemRowSet was empty: no flush needed.
I20240502 01:54:31.722016 28705 tablet.cc:1620] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b: MemRowSet was empty: no flush needed.
I20240502 01:54:31.724647 28702 tablet.cc:1722] T 69f7f6d27ad1486393b60321f8efefb3 P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:31.725135 28702 tablet.cc:1620] T 69f7f6d27ad1486393b60321f8efefb3 P c2a28baa7ee2469294d94b7fe5455cff: MemRowSet was empty: no flush needed.
I20240502 01:54:31.747175 21715 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table [id=535c7a299048459fb4fc788132d58bd7] alter complete (version 2)
I20240502 01:54:31.806349 24715 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:31.814576 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:31.839386 28689 consensus_queue.cc:1035] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:31.851949 28685 consensus_queue.cc:1035] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:31.855644 21714 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33848:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:31.865345 28701 tablet.cc:1722] T 69f7f6d27ad1486393b60321f8efefb3 P 06543b05a16f42099bc86db2a0284412: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:31.869005 28702 tablet.cc:1722] T 69f7f6d27ad1486393b60321f8efefb3 P c2a28baa7ee2469294d94b7fe5455cff: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:31.902882 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:31.903344 21714 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f9e4d9a6ca2a4b698f272d98462e62bb: rebalancer_dynamic_table_new_name [id=535c7a299048459fb4fc788132d58bd7] alter complete (version 3)
I20240502 01:54:31.907685 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:31.928978 28705 tablet.cc:1722] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:31.947863 28685 consensus_queue.cc:1035] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:31.961424 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:32.024500 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:32.044842 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 01:54:32.054167 24669 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.197:36677, user_credentials={real_user=slave}} blocked reactor thread for 79308.9us
I20240502 01:54:32.060374 28689 consensus_queue.cc:1035] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:32.112862 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:32.354127 22117 consensus_queue.cc:237] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:32.356575 22251 raft_consensus.cc:1261] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.356586 24603 raft_consensus.cc:1261] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.357151 28234 consensus_queue.cc:1035] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.357471 28229 consensus_queue.cc:1035] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.361979 28234 raft_consensus.cc:2932] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:32.363000 22251 raft_consensus.cc:2932] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:32.368144 24603 raft_consensus.cc:2932] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:32.370623 21711 catalog_manager.cc:5484] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:32.382030 21837 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bceab60366b4ade98f86072d391e7f4 with delete_type TABLET_DATA_TOMBSTONED (TS 759257ea7bef415ca698e60d82199d77 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:32.396936 28777 tablet_replica.cc:330] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:32.397203 28777 raft_consensus.cc:2229] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:32.397502 28777 raft_consensus.cc:2258] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:32.398310 28777 ts_tablet_manager.cc:1875] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:32.406980 28777 ts_tablet_manager.cc:1888] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:54:32.407119 28777 log.cc:1196] T 9bceab60366b4ade98f86072d391e7f4 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/9bceab60366b4ade98f86072d391e7f4
I20240502 01:54:32.408054 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 9bceab60366b4ade98f86072d391e7f4 (table rebalance_test_table_11 [id=e67a4afb37114119a9d514153a1563bb]) successfully deleted
I20240502 01:54:32.804781 24715 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:32.831660 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:32.832705 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:32.858414 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:32.882454 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:32.907328 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:32.978322 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:33.021742 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:33.150437 21857 consensus_queue.cc:237] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:33.152806 22117 raft_consensus.cc:1261] T 766241df5ee24264a5cb7176e6c97b91 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.152853 24734 raft_consensus.cc:1261] T 766241df5ee24264a5cb7176e6c97b91 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.153447 28560 consensus_queue.cc:1035] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:33.153759 28869 consensus_queue.cc:1035] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:33.157038 28560 raft_consensus.cc:2932] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:33.158812 22117 raft_consensus.cc:2932] T 766241df5ee24264a5cb7176e6c97b91 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:33.159662 24734 raft_consensus.cc:2932] T 766241df5ee24264a5cb7176e6c97b91 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:33.165906 21712 catalog_manager.cc:5484] T 766241df5ee24264a5cb7176e6c97b91 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:33.174232 21969 tablet_service.cc:1514] Processing DeleteTablet for tablet 766241df5ee24264a5cb7176e6c97b91 with delete_type TABLET_DATA_TOMBSTONED (TS 06543b05a16f42099bc86db2a0284412 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:33.174970 28896 tablet_replica.cc:330] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:33.175393 28896 raft_consensus.cc:2229] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:33.175637 28896 raft_consensus.cc:2258] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:33.176461 28896 ts_tablet_manager.cc:1875] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:33.181653 28896 ts_tablet_manager.cc:1888] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:54:33.181885 28896 log.cc:1196] T 766241df5ee24264a5cb7176e6c97b91 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/766241df5ee24264a5cb7176e6c97b91
I20240502 01:54:33.182725 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 766241df5ee24264a5cb7176e6c97b91 (table rebalance_test_table_5 [id=868965c86d0d413fbbeaca41748259c6]) successfully deleted
I20240502 01:54:33.596704 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:33.600911 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:33.626686 24715 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:33.649693 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:33.729732 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:33.737708 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:33.785902 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:33.800242 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:34.004005 21987 consensus_queue.cc:237] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:34.008496 21857 raft_consensus.cc:1261] T 03fb607ef0d9469daca1d6f3c5259fb2 P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:34.009341 24602 raft_consensus.cc:1261] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:34.009398 28960 consensus_queue.cc:1035] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:34.010233 28669 consensus_queue.cc:1035] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:34.017951 28669 raft_consensus.cc:2932] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:34.022135 24602 raft_consensus.cc:2932] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:34.036091 21857 raft_consensus.cc:2932] T 03fb607ef0d9469daca1d6f3c5259fb2 P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:34.043521 21712 catalog_manager.cc:5484] T 03fb607ef0d9469daca1d6f3c5259fb2 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: config changed from index 2 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New cstate: current_term: 2 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:34.059154 24715 tablet_service.cc:1514] Processing DeleteTablet for tablet 03fb607ef0d9469daca1d6f3c5259fb2 with delete_type TABLET_DATA_TOMBSTONED (TS 28157bc3e678428fb5f93c077d528513 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41650
I20240502 01:54:34.060190 29000 tablet_replica.cc:330] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:54:34.060393 29000 raft_consensus.cc:2229] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:34.060592 29000 raft_consensus.cc:2258] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:34.061262 29000 ts_tablet_manager.cc:1875] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:34.069370 29000 ts_tablet_manager.cc:1888] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 01:54:34.069630 29000 log.cc:1196] T 03fb607ef0d9469daca1d6f3c5259fb2 P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/03fb607ef0d9469daca1d6f3c5259fb2
I20240502 01:54:34.072090 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet 03fb607ef0d9469daca1d6f3c5259fb2 (table rebalancer_extra_table_0 [id=ee5b662f59f047049d4a2a52984c0cfd]) successfully deleted
I20240502 01:54:34.547019 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:34.560364 24715 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:34.572608 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:34.595865 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:34.621752 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:34.664243 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:34.691136 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:34.724700 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:34.873493 21987 consensus_queue.cc:237] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:34.894973 24867 raft_consensus.cc:1261] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:34.895653 28960 consensus_queue.cc:1035] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:34.898986 22117 raft_consensus.cc:1261] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:34.899214 28996 raft_consensus.cc:2932] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:34.899616 28960 consensus_queue.cc:1035] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:34.899835 24867 raft_consensus.cc:2932] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:34.909749 21708 catalog_manager.cc:5484] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:34.917845 22117 raft_consensus.cc:2932] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:34.936681 22229 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b6dfe0e8da4352b136d6f97d473bad with delete_type TABLET_DATA_TOMBSTONED (TS c2a28baa7ee2469294d94b7fe5455cff not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:34.938182 29099 tablet_replica.cc:330] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:34.940804 29099 raft_consensus.cc:2229] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:34.941206 29099 raft_consensus.cc:2258] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:34.942116 29099 ts_tablet_manager.cc:1875] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:34.952864 29099 ts_tablet_manager.cc:1888] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:54:34.953380 29099 log.cc:1196] T d4b6dfe0e8da4352b136d6f97d473bad P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/d4b6dfe0e8da4352b136d6f97d473bad
I20240502 01:54:34.956730 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet d4b6dfe0e8da4352b136d6f97d473bad (table rebalance_test_table_19 [id=9827c9aba79e4ba094c6e9c0e32f36f4]) successfully deleted
I20240502 01:54:35.371264 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:35.429644 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:35.431789 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:35.458493 24715 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:35.524152 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:35.550602 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:35.585337 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 01:54:35.623502 22054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.198:42375, user_credentials={real_user=slave}} blocked reactor thread for 53979.2us
I20240502 01:54:35.664345 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:36.054162 21988 consensus_queue.cc:237] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:36.062355 22118 raft_consensus.cc:1261] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:36.062781 21925 consensus_peers.cc:485] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 -> Peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Couldn't send request to peer 41c70f373c9d49a49891b03a12c6cb2b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d246b118c1c4388ac04b3775ddbee89. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:36.063097 29208 consensus_queue.cc:1035] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:36.063266 22251 raft_consensus.cc:1261] T 0d246b118c1c4388ac04b3775ddbee89 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:36.063899 28960 consensus_queue.cc:1035] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:36.074208 28960 raft_consensus.cc:2932] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:36.075204 22118 raft_consensus.cc:2932] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:36.085139 22249 raft_consensus.cc:2932] T 0d246b118c1c4388ac04b3775ddbee89 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:36.086673 21707 catalog_manager.cc:5484] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:36.128968 22118 consensus_queue.cc:237] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
W20240502 01:54:36.131852 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fb2e5d929334665ae41e33056c520bc. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:36.132185 24602 raft_consensus.cc:1261] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:36.132769 21857 raft_consensus.cc:1261] T 0fb2e5d929334665ae41e33056c520bc P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:36.133596 29186 consensus_queue.cc:1035] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:36.135525 29186 consensus_queue.cc:1035] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:36.138515 29186 raft_consensus.cc:2932] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:36.139223 24602 raft_consensus.cc:2932] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:36.146185 21714 catalog_manager.cc:5484] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:36.147946 21857 raft_consensus.cc:2932] T 0fb2e5d929334665ae41e33056c520bc P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:36.195196 25000 consensus_queue.cc:237] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:36.197786 24731 raft_consensus.cc:1261] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:36.198006 21857 raft_consensus.cc:1261] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:36.198364 24933 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76a262a323dc486f80b91c54436eb8b7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:36.198446 28975 consensus_queue.cc:1035] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:36.199038 28975 consensus_queue.cc:1035] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:36.201783 28236 raft_consensus.cc:2932] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:36.202288 21857 raft_consensus.cc:2932] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:36.206451 24731 raft_consensus.cc:2932] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:36.211880 21710 catalog_manager.cc:5484] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:36.258067 22117 consensus_queue.cc:237] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:36.261896 24731 raft_consensus.cc:1261] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:36.261898 22249 raft_consensus.cc:1261] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:36.262678 29236 consensus_queue.cc:1035] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:54:36.263427 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0eed9ee06a494588a32b4c951d6d3a14. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:36.264156 29236 consensus_queue.cc:1035] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:36.269663 29236 raft_consensus.cc:2932] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:36.270298 24731 raft_consensus.cc:2932] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:36.278362 21707 catalog_manager.cc:5484] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513 reported cstate change: config changed from index 5 to 6, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:36.282671 22249 raft_consensus.cc:2932] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:36.381996 22249 consensus_queue.cc:237] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:36.386754 21987 raft_consensus.cc:1261] T dcc8171c991b48ddbfda8007e3a87a38 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:36.386893 22117 raft_consensus.cc:1261] T dcc8171c991b48ddbfda8007e3a87a38 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:36.386914 22187 consensus_peers.cc:485] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Couldn't send request to peer 41c70f373c9d49a49891b03a12c6cb2b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcc8171c991b48ddbfda8007e3a87a38. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:36.387382 28517 consensus_queue.cc:1035] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:36.387720 27649 consensus_queue.cc:1035] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:36.402659 27649 raft_consensus.cc:2932] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:36.403442 22117 raft_consensus.cc:2932] T dcc8171c991b48ddbfda8007e3a87a38 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:36.408337 21987 raft_consensus.cc:2932] T dcc8171c991b48ddbfda8007e3a87a38 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:36.415935 21707 catalog_manager.cc:5484] T dcc8171c991b48ddbfda8007e3a87a38 P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:36.467111 24731 consensus_queue.cc:237] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:36.470441 21987 raft_consensus.cc:1261] T fa8220d5cb8140e6a66a3b377df56fe4 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:36.470800 25000 raft_consensus.cc:1261] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:36.470965 24669 consensus_peers.cc:485] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 -> Peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Couldn't send request to peer 0c64d40b5d7644bd8b29feaa80eab72f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa8220d5cb8140e6a66a3b377df56fe4. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:36.471124 29315 consensus_queue.cc:1035] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:36.471537 29068 consensus_queue.cc:1035] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:36.478190 29068 raft_consensus.cc:2932] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:36.478785 25000 raft_consensus.cc:2932] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:36.483141 21987 raft_consensus.cc:2932] T fa8220d5cb8140e6a66a3b377df56fe4 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:36.487073 21707 catalog_manager.cc:5484] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: config changed from index -1 to 2, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:36.526296 21988 consensus_queue.cc:237] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
W20240502 01:54:36.529011 21922 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b73e5695b28424ba127d1f434d1e88c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:36.529253 24603 raft_consensus.cc:1261] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:36.529448 24734 raft_consensus.cc:1261] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:36.529816 28960 consensus_queue.cc:1035] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:36.530124 29208 consensus_queue.cc:1035] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:36.537617 29335 ts_tablet_manager.cc:902] T 0fb2e5d929334665ae41e33056c520bc P 28157bc3e678428fb5f93c077d528513: Initiating tablet copy from peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:36.544394 29208 raft_consensus.cc:2932] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:36.545328 24734 raft_consensus.cc:2932] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:36.550030 24603 raft_consensus.cc:2932] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:36.552623 21707 catalog_manager.cc:5484] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:36.566305 29335 tablet_copy_client.cc:308] T 0fb2e5d929334665ae41e33056c520bc P 28157bc3e678428fb5f93c077d528513: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.195:38879
I20240502 01:54:36.568006 22140 tablet_copy_service.cc:140] P c64840657cdf4b93980bd34120c94d9a: Received BeginTabletCopySession request for tablet 0fb2e5d929334665ae41e33056c520bc from peer 28157bc3e678428fb5f93c077d528513 ({username='slave'} at 127.14.212.198:48493)
I20240502 01:54:36.568184 22140 tablet_copy_service.cc:161] P c64840657cdf4b93980bd34120c94d9a: Beginning new tablet copy session on tablet 0fb2e5d929334665ae41e33056c520bc from peer 28157bc3e678428fb5f93c077d528513 at {username='slave'} at 127.14.212.198:48493: session id = 28157bc3e678428fb5f93c077d528513-0fb2e5d929334665ae41e33056c520bc
I20240502 01:54:36.571089 22140 tablet_copy_source_session.cc:205] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:36.572459 29335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb2e5d929334665ae41e33056c520bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:36.577921 29335 tablet_copy_client.cc:783] T 0fb2e5d929334665ae41e33056c520bc P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:36.578222 29335 tablet_copy_client.cc:647] T 0fb2e5d929334665ae41e33056c520bc P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:36.580288 29335 tablet_copy_client.cc:523] T 0fb2e5d929334665ae41e33056c520bc P 28157bc3e678428fb5f93c077d528513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.583609 29335 tablet_bootstrap.cc:492] T 0fb2e5d929334665ae41e33056c520bc P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:36.605933 29335 tablet_bootstrap.cc:492] T 0fb2e5d929334665ae41e33056c520bc P 28157bc3e678428fb5f93c077d528513: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.606755 29335 tablet_bootstrap.cc:492] T 0fb2e5d929334665ae41e33056c520bc P 28157bc3e678428fb5f93c077d528513: Bootstrap complete.
I20240502 01:54:36.607033 29335 ts_tablet_manager.cc:1367] T 0fb2e5d929334665ae41e33056c520bc P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.013s
I20240502 01:54:36.607847 29335 raft_consensus.cc:348] T 0fb2e5d929334665ae41e33056c520bc P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:36.608088 29335 raft_consensus.cc:729] T 0fb2e5d929334665ae41e33056c520bc P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:36.608388 29335 consensus_queue.cc:260] T 0fb2e5d929334665ae41e33056c520bc P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:36.610229 29335 ts_tablet_manager.cc:1398] T 0fb2e5d929334665ae41e33056c520bc P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:36.610848 22140 tablet_copy_service.cc:342] P c64840657cdf4b93980bd34120c94d9a: Request end of tablet copy session 28157bc3e678428fb5f93c077d528513-0fb2e5d929334665ae41e33056c520bc received from {username='slave'} at 127.14.212.198:48493
I20240502 01:54:36.610980 22140 tablet_copy_service.cc:433] P c64840657cdf4b93980bd34120c94d9a: ending tablet copy session 28157bc3e678428fb5f93c077d528513-0fb2e5d929334665ae41e33056c520bc on tablet 0fb2e5d929334665ae41e33056c520bc with peer 28157bc3e678428fb5f93c077d528513
I20240502 01:54:36.639884 22117 consensus_queue.cc:237] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:36.643982 22249 raft_consensus.cc:1261] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 01:54:36.644555 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bceab60366b4ade98f86072d391e7f4. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:36.645579 29236 consensus_queue.cc:1035] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:36.645707 24603 raft_consensus.cc:1261] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:36.646301 29236 consensus_queue.cc:1035] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:36.650143 29233 raft_consensus.cc:2932] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:36.654862 24603 raft_consensus.cc:2932] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:36.655565 22251 raft_consensus.cc:2932] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:36.661849 21714 catalog_manager.cc:5484] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index 5 to 6, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:36.693027 29370 ts_tablet_manager.cc:902] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b: Initiating tablet copy from peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:36.701143 29370 tablet_copy_client.cc:308] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:44411
I20240502 01:54:36.701831 22010 tablet_copy_service.cc:140] P 06543b05a16f42099bc86db2a0284412: Received BeginTabletCopySession request for tablet 0d246b118c1c4388ac04b3775ddbee89 from peer 41c70f373c9d49a49891b03a12c6cb2b ({username='slave'} at 127.14.212.200:60949)
I20240502 01:54:36.701856 22249 consensus_queue.cc:237] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:36.702032 22010 tablet_copy_service.cc:161] P 06543b05a16f42099bc86db2a0284412: Beginning new tablet copy session on tablet 0d246b118c1c4388ac04b3775ddbee89 from peer 41c70f373c9d49a49891b03a12c6cb2b at {username='slave'} at 127.14.212.200:60949: session id = 41c70f373c9d49a49891b03a12c6cb2b-0d246b118c1c4388ac04b3775ddbee89
I20240502 01:54:36.705060 22120 raft_consensus.cc:1261] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:36.705127 22010 tablet_copy_source_session.cc:205] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 01:54:36.705457 22187 consensus_peers.cc:485] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Couldn't send request to peer 41c70f373c9d49a49891b03a12c6cb2b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3964f5f080f145a5923dac2f076637c8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:36.705737 28517 consensus_queue.cc:1035] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:36.705900 21988 raft_consensus.cc:1261] T 3964f5f080f145a5923dac2f076637c8 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:36.706486 29294 consensus_queue.cc:1035] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:36.710286 29370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d246b118c1c4388ac04b3775ddbee89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:36.711972 28517 raft_consensus.cc:2932] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:36.712599 22120 raft_consensus.cc:2932] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:36.719630 29370 tablet_copy_client.cc:783] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:36.720028 29370 tablet_copy_client.cc:647] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:36.719849 21713 catalog_manager.cc:5484] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:36.721653 21987 raft_consensus.cc:2932] T 3964f5f080f145a5923dac2f076637c8 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:36.731323 29370 tablet_copy_client.cc:523] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.735579 29370 tablet_bootstrap.cc:492] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:36.778239 29394 ts_tablet_manager.cc:902] T 76a262a323dc486f80b91c54436eb8b7 P 06543b05a16f42099bc86db2a0284412: Initiating tablet copy from peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:36.786067 29394 tablet_copy_client.cc:308] T 76a262a323dc486f80b91c54436eb8b7 P 06543b05a16f42099bc86db2a0284412: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.200:37097
I20240502 01:54:36.787122 25021 tablet_copy_service.cc:140] P 41c70f373c9d49a49891b03a12c6cb2b: Received BeginTabletCopySession request for tablet 76a262a323dc486f80b91c54436eb8b7 from peer 06543b05a16f42099bc86db2a0284412 ({username='slave'} at 127.14.212.194:34591)
I20240502 01:54:36.787328 25021 tablet_copy_service.cc:161] P 41c70f373c9d49a49891b03a12c6cb2b: Beginning new tablet copy session on tablet 76a262a323dc486f80b91c54436eb8b7 from peer 06543b05a16f42099bc86db2a0284412 at {username='slave'} at 127.14.212.194:34591: session id = 06543b05a16f42099bc86db2a0284412-76a262a323dc486f80b91c54436eb8b7
I20240502 01:54:36.790791 25021 tablet_copy_source_session.cc:205] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:36.797706 29370 tablet_bootstrap.cc:492] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.798321 29370 tablet_bootstrap.cc:492] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap complete.
I20240502 01:54:36.798686 29370 ts_tablet_manager.cc:1367] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.063s user 0.051s sys 0.008s
I20240502 01:54:36.799677 29370 raft_consensus.cc:348] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:36.800539 29370 raft_consensus.cc:729] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:36.801028 29370 consensus_queue.cc:260] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:36.802173 29394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a262a323dc486f80b91c54436eb8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:36.803719 29370 ts_tablet_manager.cc:1398] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:36.804605 22010 tablet_copy_service.cc:342] P 06543b05a16f42099bc86db2a0284412: Request end of tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-0d246b118c1c4388ac04b3775ddbee89 received from {username='slave'} at 127.14.212.200:60949
I20240502 01:54:36.804759 22010 tablet_copy_service.cc:433] P 06543b05a16f42099bc86db2a0284412: ending tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-0d246b118c1c4388ac04b3775ddbee89 on tablet 0d246b118c1c4388ac04b3775ddbee89 with peer 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:36.815869 29405 ts_tablet_manager.cc:902] T 0eed9ee06a494588a32b4c951d6d3a14 P d7449bb1df214bf4a5f93e4631312f08: Initiating tablet copy from peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:36.818671 29394 tablet_copy_client.cc:783] T 76a262a323dc486f80b91c54436eb8b7 P 06543b05a16f42099bc86db2a0284412: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:36.818957 29394 tablet_copy_client.cc:647] T 76a262a323dc486f80b91c54436eb8b7 P 06543b05a16f42099bc86db2a0284412: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:36.821321 29394 tablet_copy_client.cc:523] T 76a262a323dc486f80b91c54436eb8b7 P 06543b05a16f42099bc86db2a0284412: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.823557 29370 ts_tablet_manager.cc:902] T dcc8171c991b48ddbfda8007e3a87a38 P 41c70f373c9d49a49891b03a12c6cb2b: Initiating tablet copy from peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:36.826237 29370 tablet_copy_client.cc:308] T dcc8171c991b48ddbfda8007e3a87a38 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.196:43093
I20240502 01:54:36.827390 29394 tablet_bootstrap.cc:492] T 76a262a323dc486f80b91c54436eb8b7 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:36.827409 22271 tablet_copy_service.cc:140] P c2a28baa7ee2469294d94b7fe5455cff: Received BeginTabletCopySession request for tablet dcc8171c991b48ddbfda8007e3a87a38 from peer 41c70f373c9d49a49891b03a12c6cb2b ({username='slave'} at 127.14.212.200:39731)
I20240502 01:54:36.827744 22271 tablet_copy_service.cc:161] P c2a28baa7ee2469294d94b7fe5455cff: Beginning new tablet copy session on tablet dcc8171c991b48ddbfda8007e3a87a38 from peer 41c70f373c9d49a49891b03a12c6cb2b at {username='slave'} at 127.14.212.200:39731: session id = 41c70f373c9d49a49891b03a12c6cb2b-dcc8171c991b48ddbfda8007e3a87a38
I20240502 01:54:36.834290 22271 tablet_copy_source_session.cc:205] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:36.835222 29405 tablet_copy_client.cc:308] T 0eed9ee06a494588a32b4c951d6d3a14 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.195:38879
I20240502 01:54:36.835916 22140 tablet_copy_service.cc:140] P c64840657cdf4b93980bd34120c94d9a: Received BeginTabletCopySession request for tablet 0eed9ee06a494588a32b4c951d6d3a14 from peer d7449bb1df214bf4a5f93e4631312f08 ({username='slave'} at 127.14.212.199:47983)
I20240502 01:54:36.836066 22140 tablet_copy_service.cc:161] P c64840657cdf4b93980bd34120c94d9a: Beginning new tablet copy session on tablet 0eed9ee06a494588a32b4c951d6d3a14 from peer d7449bb1df214bf4a5f93e4631312f08 at {username='slave'} at 127.14.212.199:47983: session id = d7449bb1df214bf4a5f93e4631312f08-0eed9ee06a494588a32b4c951d6d3a14
I20240502 01:54:36.837749 29370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc8171c991b48ddbfda8007e3a87a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:36.839634 22140 tablet_copy_source_session.cc:205] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:36.843618 29370 tablet_copy_client.cc:783] T dcc8171c991b48ddbfda8007e3a87a38 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:36.843879 29370 tablet_copy_client.cc:647] T dcc8171c991b48ddbfda8007e3a87a38 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:36.843885 21856 consensus_queue.cc:237] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:36.846040 29370 tablet_copy_client.cc:523] T dcc8171c991b48ddbfda8007e3a87a38 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.847211 24734 raft_consensus.cc:1261] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:36.847723 24603 raft_consensus.cc:1261] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:36.850126 29370 tablet_bootstrap.cc:492] T dcc8171c991b48ddbfda8007e3a87a38 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:36.848766 29353 consensus_queue.cc:1035] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20240502 01:54:36.855990 29353 consensus_queue.cc:1035] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:54:36.857762 21790 consensus_peers.cc:485] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9f7fe67aa41b5971d5df9e4b5d788. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:36.859884 29405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eed9ee06a494588a32b4c951d6d3a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:36.887966 29394 tablet_bootstrap.cc:492] T 76a262a323dc486f80b91c54436eb8b7 P 06543b05a16f42099bc86db2a0284412: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.888598 29394 tablet_bootstrap.cc:492] T 76a262a323dc486f80b91c54436eb8b7 P 06543b05a16f42099bc86db2a0284412: Bootstrap complete.
I20240502 01:54:36.888578 29419 raft_consensus.cc:2932] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:36.889094 29394 ts_tablet_manager.cc:1367] T 76a262a323dc486f80b91c54436eb8b7 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.062s user 0.006s sys 0.002s
I20240502 01:54:36.891366 29394 raft_consensus.cc:348] T 76a262a323dc486f80b91c54436eb8b7 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:36.891597 29394 raft_consensus.cc:729] T 76a262a323dc486f80b91c54436eb8b7 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:36.897069 24731 raft_consensus.cc:2932] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:36.910508 29394 consensus_queue.cc:260] T 76a262a323dc486f80b91c54436eb8b7 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:36.911278 29405 tablet_copy_client.cc:783] T 0eed9ee06a494588a32b4c951d6d3a14 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:36.911732 29405 tablet_copy_client.cc:647] T 0eed9ee06a494588a32b4c951d6d3a14 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:36.911628 21710 catalog_manager.cc:5484] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513 reported cstate change: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:36.916818 24602 raft_consensus.cc:2932] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:36.920493 29394 ts_tablet_manager.cc:1398] T 76a262a323dc486f80b91c54436eb8b7 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.031s user 0.002s sys 0.001s
I20240502 01:54:36.920212 29405 tablet_copy_client.cc:523] T 0eed9ee06a494588a32b4c951d6d3a14 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.921272 25021 tablet_copy_service.cc:342] P 41c70f373c9d49a49891b03a12c6cb2b: Request end of tablet copy session 06543b05a16f42099bc86db2a0284412-76a262a323dc486f80b91c54436eb8b7 received from {username='slave'} at 127.14.212.194:34591
I20240502 01:54:36.921432 25021 tablet_copy_service.cc:433] P 41c70f373c9d49a49891b03a12c6cb2b: ending tablet copy session 06543b05a16f42099bc86db2a0284412-76a262a323dc486f80b91c54436eb8b7 on tablet 76a262a323dc486f80b91c54436eb8b7 with peer 06543b05a16f42099bc86db2a0284412
I20240502 01:54:36.942940 29405 tablet_bootstrap.cc:492] T 0eed9ee06a494588a32b4c951d6d3a14 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:37.024209 29437 ts_tablet_manager.cc:902] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f: Initiating tablet copy from peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:37.027318 29437 tablet_copy_client.cc:308] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.198:42375
I20240502 01:54:37.028287 24755 tablet_copy_service.cc:140] P 28157bc3e678428fb5f93c077d528513: Received BeginTabletCopySession request for tablet fa8220d5cb8140e6a66a3b377df56fe4 from peer 0c64d40b5d7644bd8b29feaa80eab72f ({username='slave'} at 127.14.212.197:50769)
I20240502 01:54:37.028556 24755 tablet_copy_service.cc:161] P 28157bc3e678428fb5f93c077d528513: Beginning new tablet copy session on tablet fa8220d5cb8140e6a66a3b377df56fe4 from peer 0c64d40b5d7644bd8b29feaa80eab72f at {username='slave'} at 127.14.212.197:50769: session id = 0c64d40b5d7644bd8b29feaa80eab72f-fa8220d5cb8140e6a66a3b377df56fe4
I20240502 01:54:37.032585 29436 ts_tablet_manager.cc:902] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77: Initiating tablet copy from peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:37.036962 24755 tablet_copy_source_session.cc:205] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:37.038655 29437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8220d5cb8140e6a66a3b377df56fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.044814 29370 tablet_bootstrap.cc:492] T dcc8171c991b48ddbfda8007e3a87a38 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:37.047494 29370 tablet_bootstrap.cc:492] T dcc8171c991b48ddbfda8007e3a87a38 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap complete.
I20240502 01:54:37.047801 29370 ts_tablet_manager.cc:1367] T dcc8171c991b48ddbfda8007e3a87a38 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.198s user 0.052s sys 0.001s
I20240502 01:54:37.048894 29370 raft_consensus.cc:348] T dcc8171c991b48ddbfda8007e3a87a38 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:37.049239 29370 raft_consensus.cc:729] T dcc8171c991b48ddbfda8007e3a87a38 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:37.049589 29370 consensus_queue.cc:260] T dcc8171c991b48ddbfda8007e3a87a38 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:37.050339 29370 ts_tablet_manager.cc:1398] T dcc8171c991b48ddbfda8007e3a87a38 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:37.051399 22271 tablet_copy_service.cc:342] P c2a28baa7ee2469294d94b7fe5455cff: Request end of tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-dcc8171c991b48ddbfda8007e3a87a38 received from {username='slave'} at 127.14.212.200:39731
I20240502 01:54:37.051537 22271 tablet_copy_service.cc:433] P c2a28baa7ee2469294d94b7fe5455cff: ending tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-dcc8171c991b48ddbfda8007e3a87a38 on tablet dcc8171c991b48ddbfda8007e3a87a38 with peer 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:37.054021 29437 tablet_copy_client.cc:783] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:37.054353 29437 tablet_copy_client.cc:647] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:37.062327 29437 tablet_copy_client.cc:523] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:37.068732 29437 tablet_bootstrap.cc:492] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:37.076054 29436 tablet_copy_client.cc:308] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:44411
I20240502 01:54:37.077175 22010 tablet_copy_service.cc:140] P 06543b05a16f42099bc86db2a0284412: Received BeginTabletCopySession request for tablet 0b73e5695b28424ba127d1f434d1e88c from peer 759257ea7bef415ca698e60d82199d77 ({username='slave'} at 127.14.212.193:54117)
I20240502 01:54:37.077412 22010 tablet_copy_service.cc:161] P 06543b05a16f42099bc86db2a0284412: Beginning new tablet copy session on tablet 0b73e5695b28424ba127d1f434d1e88c from peer 759257ea7bef415ca698e60d82199d77 at {username='slave'} at 127.14.212.193:54117: session id = 759257ea7bef415ca698e60d82199d77-0b73e5695b28424ba127d1f434d1e88c
I20240502 01:54:37.082446 22010 tablet_copy_source_session.cc:205] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:37.084295 29436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b73e5695b28424ba127d1f434d1e88c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.094564 29437 tablet_bootstrap.cc:492] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:37.094615 29405 tablet_bootstrap.cc:492] T 0eed9ee06a494588a32b4c951d6d3a14 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:37.095080 29437 tablet_bootstrap.cc:492] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap complete.
I20240502 01:54:37.095360 29437 ts_tablet_manager.cc:1367] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.000s
I20240502 01:54:37.095522 29405 tablet_bootstrap.cc:492] T 0eed9ee06a494588a32b4c951d6d3a14 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap complete.
I20240502 01:54:37.095811 29405 ts_tablet_manager.cc:1367] T 0eed9ee06a494588a32b4c951d6d3a14 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.153s user 0.048s sys 0.007s
I20240502 01:54:37.096357 29437 raft_consensus.cc:348] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:37.096616 29437 raft_consensus.cc:729] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:37.096698 29405 raft_consensus.cc:348] T 0eed9ee06a494588a32b4c951d6d3a14 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:37.096969 29405 raft_consensus.cc:729] T 0eed9ee06a494588a32b4c951d6d3a14 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:37.097273 29405 consensus_queue.cc:260] T 0eed9ee06a494588a32b4c951d6d3a14 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:37.097723 29437 consensus_queue.cc:260] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:37.098340 29437 ts_tablet_manager.cc:1398] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 01:54:37.099583 24755 tablet_copy_service.cc:342] P 28157bc3e678428fb5f93c077d528513: Request end of tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-fa8220d5cb8140e6a66a3b377df56fe4 received from {username='slave'} at 127.14.212.197:50769
I20240502 01:54:37.099741 24755 tablet_copy_service.cc:433] P 28157bc3e678428fb5f93c077d528513: ending tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-fa8220d5cb8140e6a66a3b377df56fe4 on tablet fa8220d5cb8140e6a66a3b377df56fe4 with peer 0c64d40b5d7644bd8b29feaa80eab72f
I20240502 01:54:37.100433 29405 ts_tablet_manager.cc:1398] T 0eed9ee06a494588a32b4c951d6d3a14 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:37.100713 24731 raft_consensus.cc:1203] T 0fb2e5d929334665ae41e33056c520bc P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:37.102814 22140 tablet_copy_service.cc:342] P c64840657cdf4b93980bd34120c94d9a: Request end of tablet copy session d7449bb1df214bf4a5f93e4631312f08-0eed9ee06a494588a32b4c951d6d3a14 received from {username='slave'} at 127.14.212.199:47983
I20240502 01:54:37.102931 22140 tablet_copy_service.cc:433] P c64840657cdf4b93980bd34120c94d9a: ending tablet copy session d7449bb1df214bf4a5f93e4631312f08-0eed9ee06a494588a32b4c951d6d3a14 on tablet 0eed9ee06a494588a32b4c951d6d3a14 with peer d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:37.104043 29436 tablet_copy_client.cc:783] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:37.104344 29436 tablet_copy_client.cc:647] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:37.109881 29436 tablet_copy_client.cc:523] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:37.110992 24731 consensus_queue.cc:237] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:37.114885 29436 tablet_bootstrap.cc:492] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:37.122588 24602 raft_consensus.cc:1261] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:37.122938 24668 consensus_peers.cc:485] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97c9eb341ce1470db791324a882a0271. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:37.123446 24867 raft_consensus.cc:1261] T 97c9eb341ce1470db791324a882a0271 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:37.124593 29068 consensus_queue.cc:1035] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:37.125190 29068 consensus_queue.cc:1035] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:37.126277 29436 tablet_bootstrap.cc:492] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:37.126734 29436 tablet_bootstrap.cc:492] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77: Bootstrap complete.
I20240502 01:54:37.127032 29436 ts_tablet_manager.cc:1367] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240502 01:54:37.127616 29436 raft_consensus.cc:348] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:37.127846 29436 raft_consensus.cc:729] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:37.128209 29436 consensus_queue.cc:260] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:37.128911 29436 ts_tablet_manager.cc:1398] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:37.129885 22010 tablet_copy_service.cc:342] P 06543b05a16f42099bc86db2a0284412: Request end of tablet copy session 759257ea7bef415ca698e60d82199d77-0b73e5695b28424ba127d1f434d1e88c received from {username='slave'} at 127.14.212.193:54117
I20240502 01:54:37.130029 22010 tablet_copy_service.cc:433] P 06543b05a16f42099bc86db2a0284412: ending tablet copy session 759257ea7bef415ca698e60d82199d77-0b73e5695b28424ba127d1f434d1e88c on tablet 0b73e5695b28424ba127d1f434d1e88c with peer 759257ea7bef415ca698e60d82199d77
I20240502 01:54:37.129714 29445 raft_consensus.cc:2932] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:37.142571 24865 raft_consensus.cc:2932] T 97c9eb341ce1470db791324a882a0271 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:37.142881 21710 catalog_manager.cc:5484] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 reported cstate change: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:37.143719 24603 raft_consensus.cc:2932] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:37.147768 29370 ts_tablet_manager.cc:902] T 3964f5f080f145a5923dac2f076637c8 P 41c70f373c9d49a49891b03a12c6cb2b: Initiating tablet copy from peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:37.181397 29370 tablet_copy_client.cc:308] T 3964f5f080f145a5923dac2f076637c8 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.196:43093
I20240502 01:54:37.182202 22271 tablet_copy_service.cc:140] P c2a28baa7ee2469294d94b7fe5455cff: Received BeginTabletCopySession request for tablet 3964f5f080f145a5923dac2f076637c8 from peer 41c70f373c9d49a49891b03a12c6cb2b ({username='slave'} at 127.14.212.200:39731)
I20240502 01:54:37.182344 22271 tablet_copy_service.cc:161] P c2a28baa7ee2469294d94b7fe5455cff: Beginning new tablet copy session on tablet 3964f5f080f145a5923dac2f076637c8 from peer 41c70f373c9d49a49891b03a12c6cb2b at {username='slave'} at 127.14.212.200:39731: session id = 41c70f373c9d49a49891b03a12c6cb2b-3964f5f080f145a5923dac2f076637c8
I20240502 01:54:37.184343 22271 tablet_copy_source_session.cc:205] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:37.186097 29370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3964f5f080f145a5923dac2f076637c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.192781 29370 tablet_copy_client.cc:783] T 3964f5f080f145a5923dac2f076637c8 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:37.193136 29370 tablet_copy_client.cc:647] T 3964f5f080f145a5923dac2f076637c8 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:37.196345 29370 tablet_copy_client.cc:523] T 3964f5f080f145a5923dac2f076637c8 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:37.200466 29370 tablet_bootstrap.cc:492] T 3964f5f080f145a5923dac2f076637c8 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:37.214450 24865 raft_consensus.cc:1203] T 0eed9ee06a494588a32b4c951d6d3a14 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 01:54:37.215471 25000 consensus_queue.cc:237] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
W20240502 01:54:37.221295 24936 consensus_peers.cc:485] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Couldn't send request to peer c2a28baa7ee2469294d94b7fe5455cff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee09342ee0f54863a158cb76d2112086. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:37.222574 22120 raft_consensus.cc:1261] T ee09342ee0f54863a158cb76d2112086 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:37.223667 28236 consensus_queue.cc:1035] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:37.224571 24865 raft_consensus.cc:1261] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:37.225209 28236 consensus_queue.cc:1035] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:37.242376 21987 raft_consensus.cc:1203] T 76a262a323dc486f80b91c54436eb8b7 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:37.242892 28236 raft_consensus.cc:2932] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:37.244568 22117 raft_consensus.cc:2932] T ee09342ee0f54863a158cb76d2112086 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:37.250190 24865 raft_consensus.cc:2932] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:37.255645 29370 tablet_bootstrap.cc:492] T 3964f5f080f145a5923dac2f076637c8 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:37.256255 29370 tablet_bootstrap.cc:492] T 3964f5f080f145a5923dac2f076637c8 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap complete.
I20240502 01:54:37.256649 29370 ts_tablet_manager.cc:1367] T 3964f5f080f145a5923dac2f076637c8 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.056s user 0.044s sys 0.008s
I20240502 01:54:37.257805 29370 raft_consensus.cc:348] T 3964f5f080f145a5923dac2f076637c8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:37.258245 29370 raft_consensus.cc:729] T 3964f5f080f145a5923dac2f076637c8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:37.258677 29370 consensus_queue.cc:260] T 3964f5f080f145a5923dac2f076637c8 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:37.259023 21710 catalog_manager.cc:5484] T ee09342ee0f54863a158cb76d2112086 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:37.259529 29370 ts_tablet_manager.cc:1398] T 3964f5f080f145a5923dac2f076637c8 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:37.262104 22271 tablet_copy_service.cc:342] P c2a28baa7ee2469294d94b7fe5455cff: Request end of tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-3964f5f080f145a5923dac2f076637c8 received from {username='slave'} at 127.14.212.200:39731
I20240502 01:54:37.262241 22271 tablet_copy_service.cc:433] P c2a28baa7ee2469294d94b7fe5455cff: ending tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-3964f5f080f145a5923dac2f076637c8 on tablet 3964f5f080f145a5923dac2f076637c8 with peer 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:37.266909 29405 ts_tablet_manager.cc:902] T 9bceab60366b4ade98f86072d391e7f4 P d7449bb1df214bf4a5f93e4631312f08: Initiating tablet copy from peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:54:37.285888 25000 raft_consensus.cc:1203] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:37.304598 29394 ts_tablet_manager.cc:902] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 06543b05a16f42099bc86db2a0284412: Initiating tablet copy from peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:37.340451 29405 tablet_copy_client.cc:308] T 9bceab60366b4ade98f86072d391e7f4 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.195:38879
I20240502 01:54:37.341449 22140 tablet_copy_service.cc:140] P c64840657cdf4b93980bd34120c94d9a: Received BeginTabletCopySession request for tablet 9bceab60366b4ade98f86072d391e7f4 from peer d7449bb1df214bf4a5f93e4631312f08 ({username='slave'} at 127.14.212.199:47983)
I20240502 01:54:37.341663 22140 tablet_copy_service.cc:161] P c64840657cdf4b93980bd34120c94d9a: Beginning new tablet copy session on tablet 9bceab60366b4ade98f86072d391e7f4 from peer d7449bb1df214bf4a5f93e4631312f08 at {username='slave'} at 127.14.212.199:47983: session id = d7449bb1df214bf4a5f93e4631312f08-9bceab60366b4ade98f86072d391e7f4
I20240502 01:54:37.351289 22140 tablet_copy_source_session.cc:205] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:37.352783 29394 tablet_copy_client.cc:308] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 06543b05a16f42099bc86db2a0284412: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.193:33785
I20240502 01:54:37.353413 29405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bceab60366b4ade98f86072d391e7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.356736 21878 tablet_copy_service.cc:140] P 759257ea7bef415ca698e60d82199d77: Received BeginTabletCopySession request for tablet 5ab9f7fe67aa41b5971d5df9e4b5d788 from peer 06543b05a16f42099bc86db2a0284412 ({username='slave'} at 127.14.212.194:38191)
I20240502 01:54:37.356941 21878 tablet_copy_service.cc:161] P 759257ea7bef415ca698e60d82199d77: Beginning new tablet copy session on tablet 5ab9f7fe67aa41b5971d5df9e4b5d788 from peer 06543b05a16f42099bc86db2a0284412 at {username='slave'} at 127.14.212.194:38191: session id = 06543b05a16f42099bc86db2a0284412-5ab9f7fe67aa41b5971d5df9e4b5d788
I20240502 01:54:37.360566 21878 tablet_copy_source_session.cc:205] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:37.365149 29405 tablet_copy_client.cc:783] T 9bceab60366b4ade98f86072d391e7f4 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:37.365429 29405 tablet_copy_client.cc:647] T 9bceab60366b4ade98f86072d391e7f4 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:37.367484 29405 tablet_copy_client.cc:523] T 9bceab60366b4ade98f86072d391e7f4 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:37.372368 29405 tablet_bootstrap.cc:492] T 9bceab60366b4ade98f86072d391e7f4 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:37.379674 25000 raft_consensus.cc:1203] T dcc8171c991b48ddbfda8007e3a87a38 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:37.384371 29394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab9f7fe67aa41b5971d5df9e4b5d788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.393396 29394 tablet_copy_client.cc:783] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 06543b05a16f42099bc86db2a0284412: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:37.393791 29394 tablet_copy_client.cc:647] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 06543b05a16f42099bc86db2a0284412: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:37.396018 29394 tablet_copy_client.cc:523] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 06543b05a16f42099bc86db2a0284412: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:37.409906 24602 raft_consensus.cc:1203] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:37.421511 29394 tablet_bootstrap.cc:492] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:37.451014 29394 tablet_bootstrap.cc:492] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 06543b05a16f42099bc86db2a0284412: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:37.451476 29394 tablet_bootstrap.cc:492] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 06543b05a16f42099bc86db2a0284412: Bootstrap complete.
I20240502 01:54:37.451761 29394 ts_tablet_manager.cc:1367] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.000s
I20240502 01:54:37.452548 29394 raft_consensus.cc:348] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:37.452813 29394 raft_consensus.cc:729] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:37.456024 29394 consensus_queue.cc:260] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:37.464376 29394 ts_tablet_manager.cc:1398] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240502 01:54:37.465575 21878 tablet_copy_service.cc:342] P 759257ea7bef415ca698e60d82199d77: Request end of tablet copy session 06543b05a16f42099bc86db2a0284412-5ab9f7fe67aa41b5971d5df9e4b5d788 received from {username='slave'} at 127.14.212.194:38191
I20240502 01:54:37.465713 21878 tablet_copy_service.cc:433] P 759257ea7bef415ca698e60d82199d77: ending tablet copy session 06543b05a16f42099bc86db2a0284412-5ab9f7fe67aa41b5971d5df9e4b5d788 on tablet 5ab9f7fe67aa41b5971d5df9e4b5d788 with peer 06543b05a16f42099bc86db2a0284412
I20240502 01:54:37.494091 21856 raft_consensus.cc:1203] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:37.517585 21988 consensus_queue.cc:237] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:37.517748 29405 tablet_bootstrap.cc:492] T 9bceab60366b4ade98f86072d391e7f4 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:37.518293 29405 tablet_bootstrap.cc:492] T 9bceab60366b4ade98f86072d391e7f4 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap complete.
I20240502 01:54:37.518597 29405 ts_tablet_manager.cc:1367] T 9bceab60366b4ade98f86072d391e7f4 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.146s user 0.078s sys 0.003s
I20240502 01:54:37.519456 29405 raft_consensus.cc:348] T 9bceab60366b4ade98f86072d391e7f4 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:37.519742 29405 raft_consensus.cc:729] T 9bceab60366b4ade98f86072d391e7f4 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:37.520028 29405 consensus_queue.cc:260] T 9bceab60366b4ade98f86072d391e7f4 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:37.520762 29405 ts_tablet_manager.cc:1398] T 9bceab60366b4ade98f86072d391e7f4 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:37.521668 22140 tablet_copy_service.cc:342] P c64840657cdf4b93980bd34120c94d9a: Request end of tablet copy session d7449bb1df214bf4a5f93e4631312f08-9bceab60366b4ade98f86072d391e7f4 received from {username='slave'} at 127.14.212.199:47983
I20240502 01:54:37.521816 22140 tablet_copy_service.cc:433] P c64840657cdf4b93980bd34120c94d9a: ending tablet copy session d7449bb1df214bf4a5f93e4631312f08-9bceab60366b4ade98f86072d391e7f4 on tablet 9bceab60366b4ade98f86072d391e7f4 with peer d7449bb1df214bf4a5f93e4631312f08
W20240502 01:54:37.522703 21925 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Couldn't send request to peer 41c70f373c9d49a49891b03a12c6cb2b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4b6dfe0e8da4352b136d6f97d473bad. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:37.525627 22117 raft_consensus.cc:1261] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:37.528792 24865 raft_consensus.cc:1261] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:37.529407 29511 consensus_queue.cc:1035] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:37.530099 29511 consensus_queue.cc:1035] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:37.537530 29511 raft_consensus.cc:2932] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:37.538311 24865 raft_consensus.cc:2932] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:37.545785 21714 catalog_manager.cc:5484] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index 5 to 6, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:37.546270 22117 raft_consensus.cc:2932] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:37.631975 29525 ts_tablet_manager.cc:902] T ee09342ee0f54863a158cb76d2112086 P c2a28baa7ee2469294d94b7fe5455cff: Initiating tablet copy from peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:37.650192 29525 tablet_copy_client.cc:308] T ee09342ee0f54863a158cb76d2112086 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.200:37097
I20240502 01:54:37.650584 25000 consensus_queue.cc:237] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:37.651186 25021 tablet_copy_service.cc:140] P 41c70f373c9d49a49891b03a12c6cb2b: Received BeginTabletCopySession request for tablet ee09342ee0f54863a158cb76d2112086 from peer c2a28baa7ee2469294d94b7fe5455cff ({username='slave'} at 127.14.212.196:54787)
I20240502 01:54:37.651366 25021 tablet_copy_service.cc:161] P 41c70f373c9d49a49891b03a12c6cb2b: Beginning new tablet copy session on tablet ee09342ee0f54863a158cb76d2112086 from peer c2a28baa7ee2469294d94b7fe5455cff at {username='slave'} at 127.14.212.196:54787: session id = c2a28baa7ee2469294d94b7fe5455cff-ee09342ee0f54863a158cb76d2112086
I20240502 01:54:37.654047 24731 raft_consensus.cc:1261] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:37.655498 25021 tablet_copy_source_session.cc:205] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 01:54:37.655759 24933 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 835d8839b7d84a82befac50dcc5a7c60. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:37.656016 29484 consensus_queue.cc:1035] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:37.656906 24865 raft_consensus.cc:1261] T 835d8839b7d84a82befac50dcc5a7c60 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:37.657661 29484 consensus_queue.cc:1035] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:37.660737 29525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee09342ee0f54863a158cb76d2112086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.671118 29525 tablet_copy_client.cc:783] T ee09342ee0f54863a158cb76d2112086 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:37.671526 29525 tablet_copy_client.cc:647] T ee09342ee0f54863a158cb76d2112086 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:37.674080 29525 tablet_copy_client.cc:523] T ee09342ee0f54863a158cb76d2112086 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:37.676836 29481 raft_consensus.cc:2932] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:37.678946 29525 tablet_bootstrap.cc:492] T ee09342ee0f54863a158cb76d2112086 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:37.681272 24867 raft_consensus.cc:2932] T 835d8839b7d84a82befac50dcc5a7c60 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:37.693894 21710 catalog_manager.cc:5484] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:37.697094 29525 tablet_bootstrap.cc:492] T ee09342ee0f54863a158cb76d2112086 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:37.697598 29525 tablet_bootstrap.cc:492] T ee09342ee0f54863a158cb76d2112086 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap complete.
I20240502 01:54:37.697867 29525 ts_tablet_manager.cc:1367] T ee09342ee0f54863a158cb76d2112086 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.001s
I20240502 01:54:37.700824 29525 raft_consensus.cc:348] T ee09342ee0f54863a158cb76d2112086 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:37.701283 29525 raft_consensus.cc:729] T ee09342ee0f54863a158cb76d2112086 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:37.701555 29525 consensus_queue.cc:260] T ee09342ee0f54863a158cb76d2112086 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:37.703773 24734 raft_consensus.cc:2932] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:37.704768 29525 ts_tablet_manager.cc:1398] T ee09342ee0f54863a158cb76d2112086 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:37.707928 25021 tablet_copy_service.cc:342] P 41c70f373c9d49a49891b03a12c6cb2b: Request end of tablet copy session c2a28baa7ee2469294d94b7fe5455cff-ee09342ee0f54863a158cb76d2112086 received from {username='slave'} at 127.14.212.196:54787
I20240502 01:54:37.708083 25021 tablet_copy_service.cc:433] P 41c70f373c9d49a49891b03a12c6cb2b: ending tablet copy session c2a28baa7ee2469294d94b7fe5455cff-ee09342ee0f54863a158cb76d2112086 on tablet ee09342ee0f54863a158cb76d2112086 with peer c2a28baa7ee2469294d94b7fe5455cff
I20240502 01:54:37.715991 29539 ts_tablet_manager.cc:902] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77: Initiating tablet copy from peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:37.745834 25000 raft_consensus.cc:1203] T 3964f5f080f145a5923dac2f076637c8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:37.765035 29539 tablet_copy_client.cc:308] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.198:42375
I20240502 01:54:37.766022 24755 tablet_copy_service.cc:140] P 28157bc3e678428fb5f93c077d528513: Received BeginTabletCopySession request for tablet 97c9eb341ce1470db791324a882a0271 from peer 759257ea7bef415ca698e60d82199d77 ({username='slave'} at 127.14.212.193:45457)
I20240502 01:54:37.766193 24755 tablet_copy_service.cc:161] P 28157bc3e678428fb5f93c077d528513: Beginning new tablet copy session on tablet 97c9eb341ce1470db791324a882a0271 from peer 759257ea7bef415ca698e60d82199d77 at {username='slave'} at 127.14.212.193:45457: session id = 759257ea7bef415ca698e60d82199d77-97c9eb341ce1470db791324a882a0271
I20240502 01:54:37.768626 24755 tablet_copy_source_session.cc:205] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:37.769938 29539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c9eb341ce1470db791324a882a0271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.775509 29539 tablet_copy_client.cc:783] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:37.775857 29539 tablet_copy_client.cc:647] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:37.778714 24867 raft_consensus.cc:1203] T 9bceab60366b4ade98f86072d391e7f4 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 01:54:37.778993 29539 tablet_copy_client.cc:523] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:37.783648 29539 tablet_bootstrap.cc:492] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:37.805485 29539 tablet_bootstrap.cc:492] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:37.806167 29539 tablet_bootstrap.cc:492] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77: Bootstrap complete.
I20240502 01:54:37.806634 29539 ts_tablet_manager.cc:1367] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.012s
I20240502 01:54:37.807744 29539 raft_consensus.cc:348] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:37.808135 29539 raft_consensus.cc:729] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:37.808481 29539 consensus_queue.cc:260] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:37.810051 29539 ts_tablet_manager.cc:1398] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:37.815042 24755 tablet_copy_service.cc:342] P 28157bc3e678428fb5f93c077d528513: Request end of tablet copy session 759257ea7bef415ca698e60d82199d77-97c9eb341ce1470db791324a882a0271 received from {username='slave'} at 127.14.212.193:45457
I20240502 01:54:37.815263 24755 tablet_copy_service.cc:433] P 28157bc3e678428fb5f93c077d528513: ending tablet copy session 759257ea7bef415ca698e60d82199d77-97c9eb341ce1470db791324a882a0271 on tablet 97c9eb341ce1470db791324a882a0271 with peer 759257ea7bef415ca698e60d82199d77
W20240502 01:54:37.828788 22054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.199:35737, user_credentials={real_user=slave}} blocked reactor thread for 46192.7us
I20240502 01:54:37.885728 21988 raft_consensus.cc:1203] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:37.928423 21857 consensus_queue.cc:237] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:37.931737 21988 raft_consensus.cc:1261] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:37.932422 22249 raft_consensus.cc:1261] T 84d602fa045e4cfa9b698d3e92e5cf8f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:37.935329 21791 consensus_peers.cc:485] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 -> Peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Couldn't send request to peer 0c64d40b5d7644bd8b29feaa80eab72f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84d602fa045e4cfa9b698d3e92e5cf8f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:37.935817 29419 consensus_queue.cc:1035] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:37.936175 29425 consensus_queue.cc:1035] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:37.951341 29419 raft_consensus.cc:2932] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:37.952260 22249 raft_consensus.cc:2932] T 84d602fa045e4cfa9b698d3e92e5cf8f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:37.959352 21714 catalog_manager.cc:5484] T 84d602fa045e4cfa9b698d3e92e5cf8f P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:37.980504 21988 raft_consensus.cc:2932] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:38.022275 24999 consensus_queue.cc:237] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:38.038252 24867 raft_consensus.cc:1261] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:38.038360 24602 raft_consensus.cc:1261] T d210f233014e4e80bd225256b27132a4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:38.038803 24936 consensus_peers.cc:485] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Couldn't send request to peer c2a28baa7ee2469294d94b7fe5455cff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d210f233014e4e80bd225256b27132a4. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:38.039253 29484 consensus_queue.cc:1035] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:38.039595 28236 consensus_queue.cc:1035] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:38.040544 29539 ts_tablet_manager.cc:902] T 835d8839b7d84a82befac50dcc5a7c60 P 759257ea7bef415ca698e60d82199d77: Initiating tablet copy from peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:38.043790 28236 raft_consensus.cc:2932] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:38.045723 24867 raft_consensus.cc:2932] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:38.053347 24599 raft_consensus.cc:2932] T d210f233014e4e80bd225256b27132a4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:38.057860 21714 catalog_manager.cc:5484] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:38.061213 22249 raft_consensus.cc:1203] T ee09342ee0f54863a158cb76d2112086 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:38.085589 29539 tablet_copy_client.cc:308] T 835d8839b7d84a82befac50dcc5a7c60 P 759257ea7bef415ca698e60d82199d77: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.200:37097
I20240502 01:54:38.086381 25021 tablet_copy_service.cc:140] P 41c70f373c9d49a49891b03a12c6cb2b: Received BeginTabletCopySession request for tablet 835d8839b7d84a82befac50dcc5a7c60 from peer 759257ea7bef415ca698e60d82199d77 ({username='slave'} at 127.14.212.193:44633)
I20240502 01:54:38.086550 25021 tablet_copy_service.cc:161] P 41c70f373c9d49a49891b03a12c6cb2b: Beginning new tablet copy session on tablet 835d8839b7d84a82befac50dcc5a7c60 from peer 759257ea7bef415ca698e60d82199d77 at {username='slave'} at 127.14.212.193:44633: session id = 759257ea7bef415ca698e60d82199d77-835d8839b7d84a82befac50dcc5a7c60
I20240502 01:54:38.088989 25021 tablet_copy_source_session.cc:205] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:38.090554 29539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835d8839b7d84a82befac50dcc5a7c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.099208 29539 tablet_copy_client.cc:783] T 835d8839b7d84a82befac50dcc5a7c60 P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:38.099584 29539 tablet_copy_client.cc:647] T 835d8839b7d84a82befac50dcc5a7c60 P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:38.102591 29539 tablet_copy_client.cc:523] T 835d8839b7d84a82befac50dcc5a7c60 P 759257ea7bef415ca698e60d82199d77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:38.106148 29539 tablet_bootstrap.cc:492] T 835d8839b7d84a82befac50dcc5a7c60 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:38.119572 29539 tablet_bootstrap.cc:492] T 835d8839b7d84a82befac50dcc5a7c60 P 759257ea7bef415ca698e60d82199d77: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:38.119969 29539 tablet_bootstrap.cc:492] T 835d8839b7d84a82befac50dcc5a7c60 P 759257ea7bef415ca698e60d82199d77: Bootstrap complete.
I20240502 01:54:38.120210 29539 ts_tablet_manager.cc:1367] T 835d8839b7d84a82befac50dcc5a7c60 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20240502 01:54:38.120916 29539 raft_consensus.cc:348] T 835d8839b7d84a82befac50dcc5a7c60 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:38.121129 29539 raft_consensus.cc:729] T 835d8839b7d84a82befac50dcc5a7c60 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:38.121358 29539 consensus_queue.cc:260] T 835d8839b7d84a82befac50dcc5a7c60 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:38.121948 29539 ts_tablet_manager.cc:1398] T 835d8839b7d84a82befac50dcc5a7c60 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:38.122751 25021 tablet_copy_service.cc:342] P 41c70f373c9d49a49891b03a12c6cb2b: Request end of tablet copy session 759257ea7bef415ca698e60d82199d77-835d8839b7d84a82befac50dcc5a7c60 received from {username='slave'} at 127.14.212.193:44633
I20240502 01:54:38.122905 25021 tablet_copy_service.cc:433] P 41c70f373c9d49a49891b03a12c6cb2b: ending tablet copy session 759257ea7bef415ca698e60d82199d77-835d8839b7d84a82befac50dcc5a7c60 on tablet 835d8839b7d84a82befac50dcc5a7c60 with peer 759257ea7bef415ca698e60d82199d77
I20240502 01:54:38.127745 22251 consensus_queue.cc:237] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:38.130000 21857 raft_consensus.cc:1261] T da66e78c2c5044b2924682be0b643b0a P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:38.130026 22117 raft_consensus.cc:1261] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:38.130129 22187 consensus_peers.cc:485] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff -> Peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Couldn't send request to peer 41c70f373c9d49a49891b03a12c6cb2b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da66e78c2c5044b2924682be0b643b0a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:38.130591 29384 consensus_queue.cc:1035] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:38.131052 28517 consensus_queue.cc:1035] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:38.133396 29610 ts_tablet_manager.cc:902] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b: Initiating tablet copy from peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:38.150656 29610 tablet_copy_client.cc:308] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:44411
I20240502 01:54:38.151659 22010 tablet_copy_service.cc:140] P 06543b05a16f42099bc86db2a0284412: Received BeginTabletCopySession request for tablet d4b6dfe0e8da4352b136d6f97d473bad from peer 41c70f373c9d49a49891b03a12c6cb2b ({username='slave'} at 127.14.212.200:60949)
I20240502 01:54:38.151836 22010 tablet_copy_service.cc:161] P 06543b05a16f42099bc86db2a0284412: Beginning new tablet copy session on tablet d4b6dfe0e8da4352b136d6f97d473bad from peer 41c70f373c9d49a49891b03a12c6cb2b at {username='slave'} at 127.14.212.200:60949: session id = 41c70f373c9d49a49891b03a12c6cb2b-d4b6dfe0e8da4352b136d6f97d473bad
I20240502 01:54:38.153347 29384 raft_consensus.cc:2932] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:38.154182 22010 tablet_copy_source_session.cc:205] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:38.155577 29610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b6dfe0e8da4352b136d6f97d473bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.159008 22117 raft_consensus.cc:2932] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:38.159044 21856 raft_consensus.cc:2932] T da66e78c2c5044b2924682be0b643b0a P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:38.161978 29610 tablet_copy_client.cc:783] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:38.162315 21714 catalog_manager.cc:5484] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:38.162281 29610 tablet_copy_client.cc:647] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:38.172678 29610 tablet_copy_client.cc:523] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:38.177192 29610 tablet_bootstrap.cc:492] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:38.230405 29610 tablet_bootstrap.cc:492] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:38.231657 29610 tablet_bootstrap.cc:492] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap complete.
I20240502 01:54:38.232156 29610 ts_tablet_manager.cc:1367] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.055s user 0.051s sys 0.000s
I20240502 01:54:38.233280 29610 raft_consensus.cc:348] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:38.233642 29610 raft_consensus.cc:729] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:38.233942 29610 consensus_queue.cc:260] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:38.234692 29610 ts_tablet_manager.cc:1398] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:38.235729 22010 tablet_copy_service.cc:342] P 06543b05a16f42099bc86db2a0284412: Request end of tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-d4b6dfe0e8da4352b136d6f97d473bad received from {username='slave'} at 127.14.212.200:60949
I20240502 01:54:38.235888 22010 tablet_copy_service.cc:433] P 06543b05a16f42099bc86db2a0284412: ending tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-d4b6dfe0e8da4352b136d6f97d473bad on tablet d4b6dfe0e8da4352b136d6f97d473bad with peer 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:38.242964 24734 consensus_queue.cc:237] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:38.245463 22251 raft_consensus.cc:1261] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:38.245573 24668 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b186753ceeb4a3c9f22391bb9a5b25f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:38.246057 24865 raft_consensus.cc:1261] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:38.246656 29445 consensus_queue.cc:1035] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:38.247309 29068 consensus_queue.cc:1035] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:38.255236 29068 raft_consensus.cc:2932] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:38.260334 24865 raft_consensus.cc:2932] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:38.267130 22251 raft_consensus.cc:2932] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:38.280395 21713 catalog_manager.cc:5484] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:38.328265 21856 raft_consensus.cc:1203] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:38.354707 29642 ts_tablet_manager.cc:902] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f: Initiating tablet copy from peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:38.356328 29642 tablet_copy_client.cc:308] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.193:33785
I20240502 01:54:38.357062 21878 tablet_copy_service.cc:140] P 759257ea7bef415ca698e60d82199d77: Received BeginTabletCopySession request for tablet 84d602fa045e4cfa9b698d3e92e5cf8f from peer 0c64d40b5d7644bd8b29feaa80eab72f ({username='slave'} at 127.14.212.197:36517)
I20240502 01:54:38.357219 21878 tablet_copy_service.cc:161] P 759257ea7bef415ca698e60d82199d77: Beginning new tablet copy session on tablet 84d602fa045e4cfa9b698d3e92e5cf8f from peer 0c64d40b5d7644bd8b29feaa80eab72f at {username='slave'} at 127.14.212.197:36517: session id = 0c64d40b5d7644bd8b29feaa80eab72f-84d602fa045e4cfa9b698d3e92e5cf8f
I20240502 01:54:38.359695 21878 tablet_copy_source_session.cc:205] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:38.362252 29642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d602fa045e4cfa9b698d3e92e5cf8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.368577 29642 tablet_copy_client.cc:783] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:38.368983 29642 tablet_copy_client.cc:647] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:38.371737 29642 tablet_copy_client.cc:523] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:38.376194 29642 tablet_bootstrap.cc:492] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:38.411964 24734 consensus_queue.cc:237] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:38.414937 24602 raft_consensus.cc:1261] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:38.415722 24668 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6dad8caf4f9744eea7be75d0a4890f65. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:38.415823 24999 raft_consensus.cc:1261] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:38.416455 29445 consensus_queue.cc:1035] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:38.419193 29445 consensus_queue.cc:1035] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:38.433825 29634 raft_consensus.cc:2932] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:38.434734 24603 raft_consensus.cc:2932] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:38.439333 24999 raft_consensus.cc:2932] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:38.441684 29642 tablet_bootstrap.cc:492] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:38.442180 29642 tablet_bootstrap.cc:492] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap complete.
I20240502 01:54:38.442505 29642 ts_tablet_manager.cc:1367] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.066s user 0.048s sys 0.011s
I20240502 01:54:38.443499 29642 raft_consensus.cc:348] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:38.443341 21713 catalog_manager.cc:5484] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 reported cstate change: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:38.443756 29642 raft_consensus.cc:729] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:38.444098 29642 consensus_queue.cc:260] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:38.445010 29642 ts_tablet_manager.cc:1398] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:38.445736 21878 tablet_copy_service.cc:342] P 759257ea7bef415ca698e60d82199d77: Request end of tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-84d602fa045e4cfa9b698d3e92e5cf8f received from {username='slave'} at 127.14.212.197:36517
I20240502 01:54:38.445856 21878 tablet_copy_service.cc:433] P 759257ea7bef415ca698e60d82199d77: ending tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-84d602fa045e4cfa9b698d3e92e5cf8f on tablet 84d602fa045e4cfa9b698d3e92e5cf8f with peer 0c64d40b5d7644bd8b29feaa80eab72f
I20240502 01:54:38.532784 24999 raft_consensus.cc:1203] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 01:54:38.534787 21856 raft_consensus.cc:1203] T 835d8839b7d84a82befac50dcc5a7c60 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:38.556680 29682 ts_tablet_manager.cc:902] T d210f233014e4e80bd225256b27132a4 P c2a28baa7ee2469294d94b7fe5455cff: Initiating tablet copy from peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:38.559481 29682 tablet_copy_client.cc:308] T d210f233014e4e80bd225256b27132a4 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.200:37097
I20240502 01:54:38.560334 25021 tablet_copy_service.cc:140] P 41c70f373c9d49a49891b03a12c6cb2b: Received BeginTabletCopySession request for tablet d210f233014e4e80bd225256b27132a4 from peer c2a28baa7ee2469294d94b7fe5455cff ({username='slave'} at 127.14.212.196:54787)
I20240502 01:54:38.560496 25021 tablet_copy_service.cc:161] P 41c70f373c9d49a49891b03a12c6cb2b: Beginning new tablet copy session on tablet d210f233014e4e80bd225256b27132a4 from peer c2a28baa7ee2469294d94b7fe5455cff at {username='slave'} at 127.14.212.196:54787: session id = c2a28baa7ee2469294d94b7fe5455cff-d210f233014e4e80bd225256b27132a4
I20240502 01:54:38.563627 25021 tablet_copy_source_session.cc:205] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:38.568503 29682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d210f233014e4e80bd225256b27132a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.578140 29682 tablet_copy_client.cc:783] T d210f233014e4e80bd225256b27132a4 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:38.578572 29682 tablet_copy_client.cc:647] T d210f233014e4e80bd225256b27132a4 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:38.580682 29682 tablet_copy_client.cc:523] T d210f233014e4e80bd225256b27132a4 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:38.596755 29682 tablet_bootstrap.cc:492] T d210f233014e4e80bd225256b27132a4 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:38.621948 29682 tablet_bootstrap.cc:492] T d210f233014e4e80bd225256b27132a4 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:38.622490 29682 tablet_bootstrap.cc:492] T d210f233014e4e80bd225256b27132a4 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap complete.
I20240502 01:54:38.622813 29682 ts_tablet_manager.cc:1367] T d210f233014e4e80bd225256b27132a4 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.004s
I20240502 01:54:38.623939 29682 raft_consensus.cc:348] T d210f233014e4e80bd225256b27132a4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:38.624943 29682 raft_consensus.cc:729] T d210f233014e4e80bd225256b27132a4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:38.625365 29682 consensus_queue.cc:260] T d210f233014e4e80bd225256b27132a4 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:38.626304 29682 ts_tablet_manager.cc:1398] T d210f233014e4e80bd225256b27132a4 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:38.627085 25021 tablet_copy_service.cc:342] P 41c70f373c9d49a49891b03a12c6cb2b: Request end of tablet copy session c2a28baa7ee2469294d94b7fe5455cff-d210f233014e4e80bd225256b27132a4 received from {username='slave'} at 127.14.212.196:54787
I20240502 01:54:38.627231 25021 tablet_copy_service.cc:433] P 41c70f373c9d49a49891b03a12c6cb2b: ending tablet copy session c2a28baa7ee2469294d94b7fe5455cff-d210f233014e4e80bd225256b27132a4 on tablet d210f233014e4e80bd225256b27132a4 with peer c2a28baa7ee2469294d94b7fe5455cff
I20240502 01:54:38.648084 29705 ts_tablet_manager.cc:902] T 9b186753ceeb4a3c9f22391bb9a5b25f P 759257ea7bef415ca698e60d82199d77: Initiating tablet copy from peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:38.651047 29705 tablet_copy_client.cc:308] T 9b186753ceeb4a3c9f22391bb9a5b25f P 759257ea7bef415ca698e60d82199d77: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.198:42375
I20240502 01:54:38.652012 24755 tablet_copy_service.cc:140] P 28157bc3e678428fb5f93c077d528513: Received BeginTabletCopySession request for tablet 9b186753ceeb4a3c9f22391bb9a5b25f from peer 759257ea7bef415ca698e60d82199d77 ({username='slave'} at 127.14.212.193:45457)
I20240502 01:54:38.652233 24755 tablet_copy_service.cc:161] P 28157bc3e678428fb5f93c077d528513: Beginning new tablet copy session on tablet 9b186753ceeb4a3c9f22391bb9a5b25f from peer 759257ea7bef415ca698e60d82199d77 at {username='slave'} at 127.14.212.193:45457: session id = 759257ea7bef415ca698e60d82199d77-9b186753ceeb4a3c9f22391bb9a5b25f
I20240502 01:54:38.654805 24755 tablet_copy_source_session.cc:205] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:38.656378 29705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b186753ceeb4a3c9f22391bb9a5b25f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.663972 29705 tablet_copy_client.cc:783] T 9b186753ceeb4a3c9f22391bb9a5b25f P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:38.664285 29705 tablet_copy_client.cc:647] T 9b186753ceeb4a3c9f22391bb9a5b25f P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:38.676254 29705 tablet_copy_client.cc:523] T 9b186753ceeb4a3c9f22391bb9a5b25f P 759257ea7bef415ca698e60d82199d77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:38.682695 29705 tablet_bootstrap.cc:492] T 9b186753ceeb4a3c9f22391bb9a5b25f P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:38.703838 29705 tablet_bootstrap.cc:492] T 9b186753ceeb4a3c9f22391bb9a5b25f P 759257ea7bef415ca698e60d82199d77: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:38.704417 29705 tablet_bootstrap.cc:492] T 9b186753ceeb4a3c9f22391bb9a5b25f P 759257ea7bef415ca698e60d82199d77: Bootstrap complete.
I20240502 01:54:38.704730 29705 ts_tablet_manager.cc:1367] T 9b186753ceeb4a3c9f22391bb9a5b25f P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.008s
I20240502 01:54:38.705864 29705 raft_consensus.cc:348] T 9b186753ceeb4a3c9f22391bb9a5b25f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:38.706215 29705 raft_consensus.cc:729] T 9b186753ceeb4a3c9f22391bb9a5b25f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:38.706501 29705 consensus_queue.cc:260] T 9b186753ceeb4a3c9f22391bb9a5b25f P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:38.707293 29705 ts_tablet_manager.cc:1398] T 9b186753ceeb4a3c9f22391bb9a5b25f P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:54:38.708055 24755 tablet_copy_service.cc:342] P 28157bc3e678428fb5f93c077d528513: Request end of tablet copy session 759257ea7bef415ca698e60d82199d77-9b186753ceeb4a3c9f22391bb9a5b25f received from {username='slave'} at 127.14.212.193:45457
I20240502 01:54:38.708176 24755 tablet_copy_service.cc:433] P 28157bc3e678428fb5f93c077d528513: ending tablet copy session 759257ea7bef415ca698e60d82199d77-9b186753ceeb4a3c9f22391bb9a5b25f on tablet 9b186753ceeb4a3c9f22391bb9a5b25f with peer 759257ea7bef415ca698e60d82199d77
I20240502 01:54:38.749154 29724 ts_tablet_manager.cc:902] T da66e78c2c5044b2924682be0b643b0a P 41c70f373c9d49a49891b03a12c6cb2b: Initiating tablet copy from peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:38.833333 29724 tablet_copy_client.cc:308] T da66e78c2c5044b2924682be0b643b0a P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.196:43093
I20240502 01:54:38.834494 22271 tablet_copy_service.cc:140] P c2a28baa7ee2469294d94b7fe5455cff: Received BeginTabletCopySession request for tablet da66e78c2c5044b2924682be0b643b0a from peer 41c70f373c9d49a49891b03a12c6cb2b ({username='slave'} at 127.14.212.200:39731)
I20240502 01:54:38.834676 22271 tablet_copy_service.cc:161] P c2a28baa7ee2469294d94b7fe5455cff: Beginning new tablet copy session on tablet da66e78c2c5044b2924682be0b643b0a from peer 41c70f373c9d49a49891b03a12c6cb2b at {username='slave'} at 127.14.212.200:39731: session id = 41c70f373c9d49a49891b03a12c6cb2b-da66e78c2c5044b2924682be0b643b0a
I20240502 01:54:38.837584 22271 tablet_copy_source_session.cc:205] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:38.842667 29724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da66e78c2c5044b2924682be0b643b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.863350 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:38.883762 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:38.894127 29724 tablet_copy_client.cc:783] T da66e78c2c5044b2924682be0b643b0a P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:38.894503 29724 tablet_copy_client.cc:647] T da66e78c2c5044b2924682be0b643b0a P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:38.909955 29724 tablet_copy_client.cc:523] T da66e78c2c5044b2924682be0b643b0a P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:38.921375 24715 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:38.926944 29724 tablet_bootstrap.cc:492] T da66e78c2c5044b2924682be0b643b0a P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:38.928323 24603 raft_consensus.cc:1203] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:38.970003 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:39.041901 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:39.080742 29744 ts_tablet_manager.cc:902] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412: Initiating tablet copy from peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:39.091271 21856 raft_consensus.cc:1203] T 9b186753ceeb4a3c9f22391bb9a5b25f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:39.097792 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:39.129830 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:39.154801 22251 raft_consensus.cc:1203] T d210f233014e4e80bd225256b27132a4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:39.155809 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:39.173214 29724 tablet_bootstrap.cc:492] T da66e78c2c5044b2924682be0b643b0a P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:39.173745 29724 tablet_bootstrap.cc:492] T da66e78c2c5044b2924682be0b643b0a P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap complete.
I20240502 01:54:39.174067 29724 ts_tablet_manager.cc:1367] T da66e78c2c5044b2924682be0b643b0a P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.247s user 0.065s sys 0.014s
I20240502 01:54:39.175031 29724 raft_consensus.cc:348] T da66e78c2c5044b2924682be0b643b0a P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:39.175727 29724 raft_consensus.cc:729] T da66e78c2c5044b2924682be0b643b0a P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:39.176039 29724 consensus_queue.cc:260] T da66e78c2c5044b2924682be0b643b0a P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:39.176836 29724 ts_tablet_manager.cc:1398] T da66e78c2c5044b2924682be0b643b0a P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 01:54:39.177963 22271 tablet_copy_service.cc:342] P c2a28baa7ee2469294d94b7fe5455cff: Request end of tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-da66e78c2c5044b2924682be0b643b0a received from {username='slave'} at 127.14.212.200:39731
I20240502 01:54:39.178082 22271 tablet_copy_service.cc:433] P c2a28baa7ee2469294d94b7fe5455cff: ending tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-da66e78c2c5044b2924682be0b643b0a on tablet da66e78c2c5044b2924682be0b643b0a with peer 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:39.204059 29744 tablet_copy_client.cc:308] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.198:42375
I20240502 01:54:39.204828 24755 tablet_copy_service.cc:140] P 28157bc3e678428fb5f93c077d528513: Received BeginTabletCopySession request for tablet 6dad8caf4f9744eea7be75d0a4890f65 from peer 06543b05a16f42099bc86db2a0284412 ({username='slave'} at 127.14.212.194:45255)
I20240502 01:54:39.205016 24755 tablet_copy_service.cc:161] P 28157bc3e678428fb5f93c077d528513: Beginning new tablet copy session on tablet 6dad8caf4f9744eea7be75d0a4890f65 from peer 06543b05a16f42099bc86db2a0284412 at {username='slave'} at 127.14.212.194:45255: session id = 06543b05a16f42099bc86db2a0284412-6dad8caf4f9744eea7be75d0a4890f65
I20240502 01:54:39.208364 24755 tablet_copy_source_session.cc:205] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:39.213034 29744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dad8caf4f9744eea7be75d0a4890f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:39.224786 29744 tablet_copy_client.cc:783] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:39.225209 29744 tablet_copy_client.cc:647] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:39.240489 29744 tablet_copy_client.cc:523] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:39.249622 29744 tablet_bootstrap.cc:492] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:39.286252 29744 tablet_bootstrap.cc:492] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:39.286680 29744 tablet_bootstrap.cc:492] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412: Bootstrap complete.
I20240502 01:54:39.286945 29744 ts_tablet_manager.cc:1367] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.006s
I20240502 01:54:39.287746 29744 raft_consensus.cc:348] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:39.287977 29744 raft_consensus.cc:729] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:39.288244 29744 consensus_queue.cc:260] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:39.288926 29744 ts_tablet_manager.cc:1398] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:39.289842 24755 tablet_copy_service.cc:342] P 28157bc3e678428fb5f93c077d528513: Request end of tablet copy session 06543b05a16f42099bc86db2a0284412-6dad8caf4f9744eea7be75d0a4890f65 received from {username='slave'} at 127.14.212.194:45255
I20240502 01:54:39.289979 24755 tablet_copy_service.cc:433] P 28157bc3e678428fb5f93c077d528513: ending tablet copy session 06543b05a16f42099bc86db2a0284412-6dad8caf4f9744eea7be75d0a4890f65 on tablet 6dad8caf4f9744eea7be75d0a4890f65 with peer 06543b05a16f42099bc86db2a0284412
I20240502 01:54:39.364835 25000 raft_consensus.cc:1203] T da66e78c2c5044b2924682be0b643b0a P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:39.600364 21988 raft_consensus.cc:1203] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:39.710855 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:39.730360 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:39.735457 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:39.754447 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:39.761037 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:39.761819 24715 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:39.762490 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:39.913710 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:40.033748 22249 consensus_queue.cc:237] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:40.035704 25000 raft_consensus.cc:1261] T da66e78c2c5044b2924682be0b643b0a P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:40.035724 21857 raft_consensus.cc:1261] T da66e78c2c5044b2924682be0b643b0a P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:40.036255 28517 consensus_queue.cc:1035] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:40.036492 29608 consensus_queue.cc:1035] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:40.039975 28517 raft_consensus.cc:2932] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:40.040844 25000 raft_consensus.cc:2932] T da66e78c2c5044b2924682be0b643b0a P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:40.050915 21711 catalog_manager.cc:5484] T da66e78c2c5044b2924682be0b643b0a P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:40.056432 21857 raft_consensus.cc:2932] T da66e78c2c5044b2924682be0b643b0a P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:40.073527 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet da66e78c2c5044b2924682be0b643b0a with delete_type TABLET_DATA_TOMBSTONED (TS c64840657cdf4b93980bd34120c94d9a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:40.086282 29850 tablet_replica.cc:330] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:40.086509 29850 raft_consensus.cc:2229] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:40.086824 29850 raft_consensus.cc:2258] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:40.087779 29850 ts_tablet_manager.cc:1875] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:40.092923 29850 ts_tablet_manager.cc:1888] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:54:40.093108 29850 log.cc:1196] T da66e78c2c5044b2924682be0b643b0a P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/da66e78c2c5044b2924682be0b643b0a
I20240502 01:54:40.093904 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet da66e78c2c5044b2924682be0b643b0a (table rebalance_test_table_3 [id=27064ffcfcc643d6ad082b6cd6ea21db]) successfully deleted
I20240502 01:54:40.422654 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:40.493752 24715 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:40.517153 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:40.525008 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:40.529727 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:40.532145 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:40.595104 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:40.609380 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:40.772017 24999 consensus_queue.cc:237] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:40.774389 22251 raft_consensus.cc:1261] T d210f233014e4e80bd225256b27132a4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:40.774441 24603 raft_consensus.cc:1261] T d210f233014e4e80bd225256b27132a4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:40.775033 29819 consensus_queue.cc:1035] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:40.775336 29747 consensus_queue.cc:1035] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:40.778235 29589 raft_consensus.cc:2932] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:40.779024 22251 raft_consensus.cc:2932] T d210f233014e4e80bd225256b27132a4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:40.779182 24603 raft_consensus.cc:2932] T d210f233014e4e80bd225256b27132a4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:40.787158 21706 catalog_manager.cc:5484] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:40.824199 24847 tablet_service.cc:1514] Processing DeleteTablet for tablet d210f233014e4e80bd225256b27132a4 with delete_type TABLET_DATA_TOMBSTONED (TS d7449bb1df214bf4a5f93e4631312f08 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51592
I20240502 01:54:40.839219 29948 tablet_replica.cc:330] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08: stopping tablet replica
I20240502 01:54:40.839619 29948 raft_consensus.cc:2229] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:40.840005 29948 raft_consensus.cc:2258] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:40.840732 29948 ts_tablet_manager.cc:1875] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:40.866250 29948 ts_tablet_manager.cc:1888] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:54:40.867738 29948 log.cc:1196] T d210f233014e4e80bd225256b27132a4 P d7449bb1df214bf4a5f93e4631312f08: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/wals/d210f233014e4e80bd225256b27132a4
I20240502 01:54:40.870158 21700 catalog_manager.cc:4830] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): tablet d210f233014e4e80bd225256b27132a4 (table rebalancer_extra_table_36 [id=ea4e3391af844446873822731a91ba3d]) successfully deleted
I20240502 01:54:41.152755 24715 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:41.164785 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:41.176383 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:41.187726 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:41.190675 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:41.251889 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:41.345813 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:41.358700 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:41.497283 21856 consensus_queue.cc:237] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:41.500510 24602 raft_consensus.cc:1261] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:41.501282 22251 raft_consensus.cc:1261] T 84d602fa045e4cfa9b698d3e92e5cf8f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:41.501348 29419 consensus_queue.cc:1035] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:41.501880 29419 consensus_queue.cc:1035] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:41.505962 29419 raft_consensus.cc:2932] T 84d602fa045e4cfa9b698d3e92e5cf8f P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:41.506408 22251 raft_consensus.cc:2932] T 84d602fa045e4cfa9b698d3e92e5cf8f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:41.514417 21714 catalog_manager.cc:5484] T 84d602fa045e4cfa9b698d3e92e5cf8f P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:41.538889 24602 raft_consensus.cc:2932] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:41.553580 21969 tablet_service.cc:1514] Processing DeleteTablet for tablet 84d602fa045e4cfa9b698d3e92e5cf8f with delete_type TABLET_DATA_TOMBSTONED (TS 06543b05a16f42099bc86db2a0284412 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:41.555366 30037 tablet_replica.cc:330] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:41.555584 30037 raft_consensus.cc:2229] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:41.555824 30037 raft_consensus.cc:2258] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:41.556424 30037 ts_tablet_manager.cc:1875] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:41.561740 30037 ts_tablet_manager.cc:1888] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:54:41.561913 30037 log.cc:1196] T 84d602fa045e4cfa9b698d3e92e5cf8f P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/84d602fa045e4cfa9b698d3e92e5cf8f
I20240502 01:54:41.562665 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 84d602fa045e4cfa9b698d3e92e5cf8f (table rebalance_test_table_19 [id=9827c9aba79e4ba094c6e9c0e32f36f4]) successfully deleted
I20240502 01:54:41.841037 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:41.853914 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:41.874771 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:41.883669 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:41.938800 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:41.973944 24715 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 01:54:41.976147 24933 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.193:33785, user_credentials={real_user=slave}} blocked reactor thread for 46287.2us
I20240502 01:54:42.060786 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:42.069504 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:42.208052 25001 consensus_queue.cc:237] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:42.210362 24867 raft_consensus.cc:1261] T 835d8839b7d84a82befac50dcc5a7c60 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:42.210558 21857 raft_consensus.cc:1261] T 835d8839b7d84a82befac50dcc5a7c60 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:42.211040 29819 consensus_queue.cc:1035] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:42.211352 30092 consensus_queue.cc:1035] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:42.218158 29819 raft_consensus.cc:2932] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:42.219471 21857 raft_consensus.cc:2932] T 835d8839b7d84a82befac50dcc5a7c60 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:42.225171 24867 raft_consensus.cc:2932] T 835d8839b7d84a82befac50dcc5a7c60 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:42.227504 21710 catalog_manager.cc:5484] T 835d8839b7d84a82befac50dcc5a7c60 P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:42.241261 24715 tablet_service.cc:1514] Processing DeleteTablet for tablet 835d8839b7d84a82befac50dcc5a7c60 with delete_type TABLET_DATA_TOMBSTONED (TS 28157bc3e678428fb5f93c077d528513 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41650
I20240502 01:54:42.241971 30120 tablet_replica.cc:330] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:54:42.242235 30120 raft_consensus.cc:2229] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:42.242614 30120 raft_consensus.cc:2258] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:42.243374 30120 ts_tablet_manager.cc:1875] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:42.250520 30120 ts_tablet_manager.cc:1888] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:54:42.250641 30120 log.cc:1196] T 835d8839b7d84a82befac50dcc5a7c60 P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/835d8839b7d84a82befac50dcc5a7c60
I20240502 01:54:42.251370 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet 835d8839b7d84a82befac50dcc5a7c60 (table rebalancer_extra_table_10 [id=543ab4cdef864690bedccba59308edc5]) successfully deleted
I20240502 01:54:42.583086 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:42.607496 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:42.609242 24715 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:42.623318 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:42.641147 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:42.675748 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:42.683501 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:42.714807 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 01:54:42.722888 24802 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.198:42375, user_credentials={real_user=slave}} blocked reactor thread for 59584.7us
I20240502 01:54:42.874111 21986 consensus_queue.cc:237] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:42.878010 24866 raft_consensus.cc:1261] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:42.878062 25001 raft_consensus.cc:1261] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:42.878710 28960 consensus_queue.cc:1035] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:42.879174 29900 consensus_queue.cc:1035] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:42.886139 29900 raft_consensus.cc:2932] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:42.889173 24999 raft_consensus.cc:2932] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:42.904620 24866 raft_consensus.cc:2932] T d4b6dfe0e8da4352b136d6f97d473bad P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:42.906054 21710 catalog_manager.cc:5484] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index 6 to 7, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:42.953979 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b6dfe0e8da4352b136d6f97d473bad with delete_type TABLET_DATA_TOMBSTONED (TS c64840657cdf4b93980bd34120c94d9a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:42.965407 30214 tablet_replica.cc:330] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:42.965719 30214 raft_consensus.cc:2229] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:42.965974 30214 raft_consensus.cc:2258] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:42.966836 30214 ts_tablet_manager.cc:1875] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:42.973008 30214 ts_tablet_manager.cc:1888] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:54:42.973273 30214 log.cc:1196] T d4b6dfe0e8da4352b136d6f97d473bad P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/d4b6dfe0e8da4352b136d6f97d473bad
I20240502 01:54:42.974262 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet d4b6dfe0e8da4352b136d6f97d473bad (table rebalance_test_table_19 [id=9827c9aba79e4ba094c6e9c0e32f36f4]) successfully deleted
I20240502 01:54:43.210850 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:43.212852 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:43.233834 24715 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:43.278898 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:43.364779 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:43.374477 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:43.397601 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:43.491226 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:43.580389 24732 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b186753ceeb4a3c9f22391bb9a5b25f"
dest_uuid: "28157bc3e678428fb5f93c077d528513"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43242
I20240502 01:54:43.580618 24732 raft_consensus.cc:595] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:43.803328 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:43.851766 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:43.868235 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:43.876173 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:43.896940 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:43.915481 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 01:54:43.976817 24669 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.197:36677, user_credentials={real_user=slave}} blocked reactor thread for 95615.3us
I20240502 01:54:43.994741 24715 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:44.032692 30324 raft_consensus.cc:979] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513: : Instructing follower c2a28baa7ee2469294d94b7fe5455cff to start an election
I20240502 01:54:44.032857 30324 raft_consensus.cc:1067] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Signalling peer c2a28baa7ee2469294d94b7fe5455cff to start an election
I20240502 01:54:44.033843 22249 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9b186753ceeb4a3c9f22391bb9a5b25f"
dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
from {username='slave'} at 127.14.212.198:56765
I20240502 01:54:44.034147 22249 raft_consensus.cc:482] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:44.034263 22249 raft_consensus.cc:3037] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:44.038415 22249 raft_consensus.cc:504] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:44.040726 24866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b186753ceeb4a3c9f22391bb9a5b25f" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "d7449bb1df214bf4a5f93e4631312f08"
I20240502 01:54:44.040982 24866 raft_consensus.cc:3037] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:44.041013 22249 leader_election.cc:288] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:44.041488 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b186753ceeb4a3c9f22391bb9a5b25f" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:44.041685 21856 raft_consensus.cc:3037] T 9b186753ceeb4a3c9f22391bb9a5b25f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:44.043673 24866 raft_consensus.cc:2454] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 2.
I20240502 01:54:44.044337 24734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b186753ceeb4a3c9f22391bb9a5b25f" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:44.044524 24734 raft_consensus.cc:3032] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:44.044579 24734 raft_consensus.cc:729] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:44.044420 21856 raft_consensus.cc:2454] T 9b186753ceeb4a3c9f22391bb9a5b25f P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 2.
I20240502 01:54:44.044826 24734 consensus_queue.cc:260] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:44.045233 24734 raft_consensus.cc:3037] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:44.045953 22184 leader_election.cc:302] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff, d7449bb1df214bf4a5f93e4631312f08; no voters:
I20240502 01:54:44.046240 30262 raft_consensus.cc:2781] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:44.046353 30262 raft_consensus.cc:686] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [term 2 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:54:44.046614 30262 consensus_queue.cc:237] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:44.048445 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:44.048933 24734 raft_consensus.cc:2454] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 2.
I20240502 01:54:44.098574 21707 catalog_manager.cc:5484] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 1 to 2, leader changed from 28157bc3e678428fb5f93c077d528513 (127.14.212.198) to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 2 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
W20240502 01:54:44.137605 22186 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.195:38879, user_credentials={real_user=slave}} blocked reactor thread for 43613.6us
I20240502 01:54:44.517419 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:44.561959 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:44.565871 24866 raft_consensus.cc:1261] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08 [term 2 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:54:44.566448 30338 consensus_queue.cc:1035] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:44.604984 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:44.610924 24732 raft_consensus.cc:1261] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:54:44.611871 30337 consensus_queue.cc:1035] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:44.618335 21854 raft_consensus.cc:1261] T 9b186753ceeb4a3c9f22391bb9a5b25f P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:54:44.621193 30337 consensus_queue.cc:1035] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:44.675242 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:44.692407 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:44.710141 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:44.714319 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:44.728367 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:44.915935 24999 consensus_queue.cc:237] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:44.918187 22249 raft_consensus.cc:1261] T ee09342ee0f54863a158cb76d2112086 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:44.918473 22118 raft_consensus.cc:1261] T ee09342ee0f54863a158cb76d2112086 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:44.919049 30331 consensus_queue.cc:1035] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:44.922456 30331 consensus_queue.cc:1035] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:44.937554 29819 raft_consensus.cc:2932] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:44.938146 22249 raft_consensus.cc:2932] T ee09342ee0f54863a158cb76d2112086 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:44.939179 22118 raft_consensus.cc:2932] T ee09342ee0f54863a158cb76d2112086 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:44.946379 21706 catalog_manager.cc:5484] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:44.967403 24847 tablet_service.cc:1514] Processing DeleteTablet for tablet ee09342ee0f54863a158cb76d2112086 with delete_type TABLET_DATA_TOMBSTONED (TS d7449bb1df214bf4a5f93e4631312f08 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51592
I20240502 01:54:44.973570 30450 tablet_replica.cc:330] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08: stopping tablet replica
I20240502 01:54:44.973799 30450 raft_consensus.cc:2229] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:44.973997 30450 raft_consensus.cc:2258] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:44.974592 30450 ts_tablet_manager.cc:1875] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:44.980195 30450 ts_tablet_manager.cc:1888] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:54:44.980293 30450 log.cc:1196] T ee09342ee0f54863a158cb76d2112086 P d7449bb1df214bf4a5f93e4631312f08: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/wals/ee09342ee0f54863a158cb76d2112086
I20240502 01:54:44.981061 21700 catalog_manager.cc:4830] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): tablet ee09342ee0f54863a158cb76d2112086 (table rebalancer_extra_table_28 [id=3bb44c54e77145c39bccee47a8077319]) successfully deleted
I20240502 01:54:45.229235 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:45.251053 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:45.261176 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:45.321691 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:45.355003 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:45.386657 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:45.445062 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:45.484426 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:45.661300 24734 consensus_queue.cc:237] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:45.663693 21854 raft_consensus.cc:1261] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:45.664273 24867 raft_consensus.cc:1261] T 97c9eb341ce1470db791324a882a0271 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:45.664258 30010 consensus_queue.cc:1035] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:45.664755 30413 consensus_queue.cc:1035] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:45.668304 30010 raft_consensus.cc:2932] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:45.668917 21854 raft_consensus.cc:2932] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:45.674439 21713 catalog_manager.cc:5484] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:45.676870 24867 raft_consensus.cc:2932] T 97c9eb341ce1470db791324a882a0271 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:45.697479 24583 tablet_service.cc:1514] Processing DeleteTablet for tablet 97c9eb341ce1470db791324a882a0271 with delete_type TABLET_DATA_TOMBSTONED (TS 0c64d40b5d7644bd8b29feaa80eab72f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53832
I20240502 01:54:45.698177 30545 tablet_replica.cc:330] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:54:45.698467 30545 raft_consensus.cc:2229] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:45.699255 30545 raft_consensus.cc:2258] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:45.700052 30545 ts_tablet_manager.cc:1875] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:45.705276 30545 ts_tablet_manager.cc:1888] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:54:45.705374 30545 log.cc:1196] T 97c9eb341ce1470db791324a882a0271 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/97c9eb341ce1470db791324a882a0271
I20240502 01:54:45.708599 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet 97c9eb341ce1470db791324a882a0271 (table rebalancer_extra_table_36 [id=ea4e3391af844446873822731a91ba3d]) successfully deleted
I20240502 01:54:46.063722 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:46.105074 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:46.125190 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:46.132619 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:46.136723 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:46.199457 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:46.226716 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:46.272365 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:46.434557 21854 consensus_queue.cc:237] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:46.436944 24603 raft_consensus.cc:1261] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:46.437148 21986 raft_consensus.cc:1261] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:46.437579 30600 consensus_queue.cc:1035] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:46.437860 30335 consensus_queue.cc:1035] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:46.441303 30600 raft_consensus.cc:2932] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:46.441951 21986 raft_consensus.cc:2932] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:46.442374 24603 raft_consensus.cc:2932] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:46.449999 21713 catalog_manager.cc:5484] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:46.459812 24715 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ab9f7fe67aa41b5971d5df9e4b5d788 with delete_type TABLET_DATA_TOMBSTONED (TS 28157bc3e678428fb5f93c077d528513 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41650
I20240502 01:54:46.460436 30628 tablet_replica.cc:330] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:54:46.460644 30628 raft_consensus.cc:2229] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:46.460873 30628 raft_consensus.cc:2258] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:46.461691 30628 ts_tablet_manager.cc:1875] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:46.467659 30628 ts_tablet_manager.cc:1888] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:54:46.467774 30628 log.cc:1196] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/5ab9f7fe67aa41b5971d5df9e4b5d788
I20240502 01:54:46.468631 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet 5ab9f7fe67aa41b5971d5df9e4b5d788 (table rebalancer_extra_table_12 [id=091ceb1644c44e13b8f3ed4e93da3341]) successfully deleted
I20240502 01:54:46.770660 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:46.772563 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:46.813714 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:46.831238 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:46.841543 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:46.849685 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:46.918171 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:46.974687 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:47.109130 22118 consensus_queue.cc:237] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:47.111243 24867 raft_consensus.cc:1261] T 9bceab60366b4ade98f86072d391e7f4 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:47.111292 24603 raft_consensus.cc:1261] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:47.111908 29186 consensus_queue.cc:1035] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:47.112622 29186 consensus_queue.cc:1035] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:47.119791 29186 raft_consensus.cc:2932] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:47.121623 24867 raft_consensus.cc:2932] T 9bceab60366b4ade98f86072d391e7f4 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:47.127835 21714 catalog_manager.cc:5484] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index 6 to 7, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:47.144202 22229 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bceab60366b4ade98f86072d391e7f4 with delete_type TABLET_DATA_TOMBSTONED (TS c2a28baa7ee2469294d94b7fe5455cff not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:47.145344 30717 tablet_replica.cc:330] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:47.145681 30717 raft_consensus.cc:2229] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:47.145504 24602 raft_consensus.cc:2932] T 9bceab60366b4ade98f86072d391e7f4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:47.145961 30717 raft_consensus.cc:2258] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:47.146831 30717 ts_tablet_manager.cc:1875] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:47.153296 30717 ts_tablet_manager.cc:1888] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:54:47.153549 30717 log.cc:1196] T 9bceab60366b4ade98f86072d391e7f4 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/9bceab60366b4ade98f86072d391e7f4
I20240502 01:54:47.154543 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 9bceab60366b4ade98f86072d391e7f4 (table rebalance_test_table_11 [id=e67a4afb37114119a9d514153a1563bb]) successfully deleted
I20240502 01:54:47.551221 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:47.581646 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:47.626060 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:47.655923 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:47.660766 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:47.695714 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:47.702394 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:47.717133 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:47.914076 22249 consensus_queue.cc:237] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:47.920610 24999 raft_consensus.cc:1261] T 3964f5f080f145a5923dac2f076637c8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:47.921700 21986 raft_consensus.cc:1261] T 3964f5f080f145a5923dac2f076637c8 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:47.924489 30765 consensus_queue.cc:1035] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:47.927506 30765 consensus_queue.cc:1035] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:47.930389 30337 raft_consensus.cc:2932] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:47.933674 21986 raft_consensus.cc:2932] T 3964f5f080f145a5923dac2f076637c8 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:47.939465 21714 catalog_manager.cc:5484] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:47.956761 24999 raft_consensus.cc:2932] T 3964f5f080f145a5923dac2f076637c8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:47.973565 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 3964f5f080f145a5923dac2f076637c8 with delete_type TABLET_DATA_TOMBSTONED (TS c64840657cdf4b93980bd34120c94d9a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:47.974238 30814 tablet_replica.cc:330] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:47.974530 30814 raft_consensus.cc:2229] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:47.975049 30814 raft_consensus.cc:2258] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:47.975960 30814 ts_tablet_manager.cc:1875] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:47.984179 30814 ts_tablet_manager.cc:1888] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:54:47.984313 30814 log.cc:1196] T 3964f5f080f145a5923dac2f076637c8 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/3964f5f080f145a5923dac2f076637c8
I20240502 01:54:47.985193 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 3964f5f080f145a5923dac2f076637c8 (table rebalance_test_table_11 [id=e67a4afb37114119a9d514153a1563bb]) successfully deleted
I20240502 01:54:48.257694 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:48.410348 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:48.418227 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:48.438426 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:48.478576 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:48.490533 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:48.504928 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:48.517467 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:48.668143 21986 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0b73e5695b28424ba127d1f434d1e88c"
dest_uuid: "06543b05a16f42099bc86db2a0284412"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57388
I20240502 01:54:48.668391 21986 raft_consensus.cc:595] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:48.858613 30875 raft_consensus.cc:979] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412: : Instructing follower 28157bc3e678428fb5f93c077d528513 to start an election
I20240502 01:54:48.858803 30875 raft_consensus.cc:1067] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Signalling peer 28157bc3e678428fb5f93c077d528513 to start an election
I20240502 01:54:48.859721 24734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0b73e5695b28424ba127d1f434d1e88c"
dest_uuid: "28157bc3e678428fb5f93c077d528513"
from {username='slave'} at 127.14.212.194:45255
I20240502 01:54:48.859966 24734 raft_consensus.cc:482] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:48.860060 24734 raft_consensus.cc:3037] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.864096 24734 raft_consensus.cc:504] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:48.867460 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b73e5695b28424ba127d1f434d1e88c" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:54:48.867720 21986 raft_consensus.cc:3032] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:48.867810 21986 raft_consensus.cc:729] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:54:48.867767 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b73e5695b28424ba127d1f434d1e88c" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:54:48.867964 24603 raft_consensus.cc:3037] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.868170 21986 consensus_queue.cc:260] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:48.868640 21986 raft_consensus.cc:3037] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.870718 24603 raft_consensus.cc:2454] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 2.
I20240502 01:54:48.871344 21986 raft_consensus.cc:2454] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 2.
I20240502 01:54:48.871843 24668 leader_election.cc:302] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:54:48.872413 30761 raft_consensus.cc:2781] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:48.872577 30761 raft_consensus.cc:686] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [term 2 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:48.872864 30761 consensus_queue.cc:237] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:48.874044 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b73e5695b28424ba127d1f434d1e88c" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:54:48.874280 21856 raft_consensus.cc:3037] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.874945 24734 leader_election.cc:288] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:48.877549 21856 raft_consensus.cc:2454] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 2.
I20240502 01:54:48.892269 21712 catalog_manager.cc:5484] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 1 to 2, leader changed from 06543b05a16f42099bc86db2a0284412 (127.14.212.194) to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 2 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:49.040673 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:54:49.044046 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:49.094810 24715 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:49.130040 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:49.194880 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:49.208484 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:49.227938 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:49.251468 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:49.276106 24602 raft_consensus.cc:1261] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:54:49.276700 30761 consensus_queue.cc:1035] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:49.347539 21856 raft_consensus.cc:1261] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:54:49.348295 30869 consensus_queue.cc:1035] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:49.374989 21986 raft_consensus.cc:1261] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:54:49.377218 30948 consensus_queue.cc:1035] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:49.726011 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:54:49.749830 24715 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:49.750162 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:49.769215 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:49.774009 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:49.825099 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:49.838675 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 01:54:49.925554 24668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.194:44411, user_credentials={real_user=slave}} blocked reactor thread for 46277.4us
I20240502 01:54:49.938688 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:50.052861 22249 consensus_queue.cc:237] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:50.055063 21987 raft_consensus.cc:1261] T dcc8171c991b48ddbfda8007e3a87a38 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:50.055106 24999 raft_consensus.cc:1261] T dcc8171c991b48ddbfda8007e3a87a38 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:50.055632 30493 consensus_queue.cc:1035] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:50.055939 30918 consensus_queue.cc:1035] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:50.058611 30493 raft_consensus.cc:2932] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:50.059161 21987 raft_consensus.cc:2932] T dcc8171c991b48ddbfda8007e3a87a38 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:50.059259 24999 raft_consensus.cc:2932] T dcc8171c991b48ddbfda8007e3a87a38 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:50.065407 21708 catalog_manager.cc:5484] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:50.073406 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc8171c991b48ddbfda8007e3a87a38 with delete_type TABLET_DATA_TOMBSTONED (TS c64840657cdf4b93980bd34120c94d9a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:50.075196 31046 tablet_replica.cc:330] T dcc8171c991b48ddbfda8007e3a87a38 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:50.075551 31046 raft_consensus.cc:2229] T dcc8171c991b48ddbfda8007e3a87a38 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:50.075917 31046 raft_consensus.cc:2258] T dcc8171c991b48ddbfda8007e3a87a38 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:50.076930 31046 ts_tablet_manager.cc:1875] T dcc8171c991b48ddbfda8007e3a87a38 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:50.083854 31046 ts_tablet_manager.cc:1888] T dcc8171c991b48ddbfda8007e3a87a38 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:54:50.083979 31046 log.cc:1196] T dcc8171c991b48ddbfda8007e3a87a38 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/dcc8171c991b48ddbfda8007e3a87a38
I20240502 01:54:50.084715 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet dcc8171c991b48ddbfda8007e3a87a38 (table rebalance_test_table_1 [id=eaf29e0201fc4b738b1b30c13a773ffd]) successfully deleted
I20240502 01:54:50.473242 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:54:50.502636 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:50.517259 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:50.518615 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:50.545756 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:50.610539 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:50.629156 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:50.661412 24715 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:50.777801 22118 consensus_queue.cc:237] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:50.780292 24734 raft_consensus.cc:1261] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:50.781026 30785 consensus_queue.cc:1035] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:50.784051 24866 raft_consensus.cc:1261] T 0eed9ee06a494588a32b4c951d6d3a14 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:50.784505 30785 raft_consensus.cc:2932] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:50.785959 31080 consensus_queue.cc:1035] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:50.786206 24734 raft_consensus.cc:2932] T 0eed9ee06a494588a32b4c951d6d3a14 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:50.791623 21712 catalog_manager.cc:5484] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index 6 to 7, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:50.793026 24866 raft_consensus.cc:2932] T 0eed9ee06a494588a32b4c951d6d3a14 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:50.823858 22229 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eed9ee06a494588a32b4c951d6d3a14 with delete_type TABLET_DATA_TOMBSTONED (TS c2a28baa7ee2469294d94b7fe5455cff not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50898
I20240502 01:54:50.826189 31140 tablet_replica.cc:330] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:54:50.826411 31140 raft_consensus.cc:2229] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:50.826763 31140 raft_consensus.cc:2258] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:50.827584 31140 ts_tablet_manager.cc:1875] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:50.836131 31140 ts_tablet_manager.cc:1888] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:54:50.836392 31140 log.cc:1196] T 0eed9ee06a494588a32b4c951d6d3a14 P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/0eed9ee06a494588a32b4c951d6d3a14
I20240502 01:54:50.837172 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 0eed9ee06a494588a32b4c951d6d3a14 (table rebalance_test_table_1 [id=eaf29e0201fc4b738b1b30c13a773ffd]) successfully deleted
I20240502 01:54:51.101392 24583 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:51.101812 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:54:51.148483 24715 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:51.182045 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:51.238022 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:51.238302 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:51.334574 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:51.364028 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:51.508425 24734 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fa8220d5cb8140e6a66a3b377df56fe4"
dest_uuid: "28157bc3e678428fb5f93c077d528513"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42974
I20240502 01:54:51.508685 24734 raft_consensus.cc:595] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:51.525022 30949 raft_consensus.cc:979] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513: : Instructing follower 0c64d40b5d7644bd8b29feaa80eab72f to start an election
I20240502 01:54:51.525203 30949 raft_consensus.cc:1067] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Signalling peer 0c64d40b5d7644bd8b29feaa80eab72f to start an election
I20240502 01:54:51.526268 24601 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fa8220d5cb8140e6a66a3b377df56fe4"
dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
from {username='slave'} at 127.14.212.198:35347
I20240502 01:54:51.526562 24601 raft_consensus.cc:482] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:51.526691 24601 raft_consensus.cc:3037] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:51.529429 24601 raft_consensus.cc:504] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:51.531065 21987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8220d5cb8140e6a66a3b377df56fe4" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:54:51.531315 21987 raft_consensus.cc:3037] T fa8220d5cb8140e6a66a3b377df56fe4 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:51.534046 21987 raft_consensus.cc:2454] T fa8220d5cb8140e6a66a3b377df56fe4 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 2.
I20240502 01:54:51.537703 24601 leader_election.cc:288] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:51.538229 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8220d5cb8140e6a66a3b377df56fe4" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:54:51.538414 24732 raft_consensus.cc:3032] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:51.538486 24732 raft_consensus.cc:729] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:54:51.538710 24732 consensus_queue.cc:260] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:51.539024 24732 raft_consensus.cc:3037] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:51.540768 24732 raft_consensus.cc:2454] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 2.
I20240502 01:54:51.541203 24537 leader_election.cc:302] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:54:51.541332 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8220d5cb8140e6a66a3b377df56fe4" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:54:51.541477 31090 raft_consensus.cc:2781] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:51.541545 25000 raft_consensus.cc:3037] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:51.541661 31090 raft_consensus.cc:686] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:54:51.541945 31090 consensus_queue.cc:237] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:51.543910 25000 raft_consensus.cc:2454] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 2.
I20240502 01:54:51.546846 21707 catalog_manager.cc:5484] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 1 to 2, leader changed from 28157bc3e678428fb5f93c077d528513 (127.14.212.198) to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 2 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:51.778393 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:51.780382 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:54:51.785286 24583 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:51.808735 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:51.831713 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:51.849656 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:51.857626 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:51.864753 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:52.039889 24732 raft_consensus.cc:1261] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:54:52.040596 30786 consensus_queue.cc:1035] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:52.041263 21986 raft_consensus.cc:1261] T fa8220d5cb8140e6a66a3b377df56fe4 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:54:52.042059 30786 consensus_queue.cc:1035] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:52.063746 24999 raft_consensus.cc:1261] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:54:52.064658 30786 consensus_queue.cc:1035] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:52.363190 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:54:52.384404 24583 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:52.416532 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:52.427145 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:52.526782 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:52.578233 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:52.593119 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:52.628123 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:52.739678 25000 consensus_queue.cc:237] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:52.741936 24734 raft_consensus.cc:1261] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:52.742492 21987 raft_consensus.cc:1261] T 76a262a323dc486f80b91c54436eb8b7 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:52.742497 29819 consensus_queue.cc:1035] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:52.743089 29819 consensus_queue.cc:1035] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:52.745296 30092 raft_consensus.cc:2932] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:52.745949 24734 raft_consensus.cc:2932] T 76a262a323dc486f80b91c54436eb8b7 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:52.753070 21987 raft_consensus.cc:2932] T 76a262a323dc486f80b91c54436eb8b7 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:52.754786 21712 catalog_manager.cc:5484] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:52.772205 21837 tablet_service.cc:1514] Processing DeleteTablet for tablet 76a262a323dc486f80b91c54436eb8b7 with delete_type TABLET_DATA_TOMBSTONED (TS 759257ea7bef415ca698e60d82199d77 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47656
I20240502 01:54:52.774482 31353 tablet_replica.cc:330] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:54:52.774811 31353 raft_consensus.cc:2229] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:52.775135 31353 raft_consensus.cc:2258] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:52.775796 31353 ts_tablet_manager.cc:1875] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:52.781102 31353 ts_tablet_manager.cc:1888] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:54:52.781229 31353 log.cc:1196] T 76a262a323dc486f80b91c54436eb8b7 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/76a262a323dc486f80b91c54436eb8b7
I20240502 01:54:52.782140 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 76a262a323dc486f80b91c54436eb8b7 (table rebalancer_extra_table_2 [id=06ae89337ff143b5a107c994cbecc25a]) successfully deleted
I20240502 01:54:53.083330 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:54:53.148898 24583 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:53.150712 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:53.159241 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:53.198068 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:53.225979 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:53.244401 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:53.268478 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:53.496315 22117 consensus_queue.cc:237] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:53.501689 21856 raft_consensus.cc:1261] T 0fb2e5d929334665ae41e33056c520bc P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:53.502247 31080 consensus_queue.cc:1035] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:53.502466 24731 raft_consensus.cc:1261] T 0fb2e5d929334665ae41e33056c520bc P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:53.503202 31080 consensus_queue.cc:1035] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:53.507109 31080 raft_consensus.cc:2932] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:53.513876 21856 raft_consensus.cc:2932] T 0fb2e5d929334665ae41e33056c520bc P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:53.520562 24731 raft_consensus.cc:2932] T 0fb2e5d929334665ae41e33056c520bc P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:53.521613 21715 catalog_manager.cc:5484] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:53.595381 24583 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb2e5d929334665ae41e33056c520bc with delete_type TABLET_DATA_TOMBSTONED (TS 0c64d40b5d7644bd8b29feaa80eab72f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53832
I20240502 01:54:53.625409 31438 tablet_replica.cc:330] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:54:53.625705 31438 raft_consensus.cc:2229] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:53.625959 31438 raft_consensus.cc:2258] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:53.627358 31438 ts_tablet_manager.cc:1875] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:53.643031 31438 ts_tablet_manager.cc:1888] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:54:53.643285 31438 log.cc:1196] T 0fb2e5d929334665ae41e33056c520bc P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/0fb2e5d929334665ae41e33056c520bc
I20240502 01:54:53.645505 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet 0fb2e5d929334665ae41e33056c520bc (table rebalancer_extra_table_32 [id=e6f275d6ff034b158e43e06075db0828]) successfully deleted
I20240502 01:54:53.912457 24583 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:53.929948 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:54:53.967097 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:54.008625 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:54.077131 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:54.093937 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:54.100802 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:54.122197 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:54.234153 21987 consensus_queue.cc:237] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:54.236249 25000 raft_consensus.cc:1261] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:54.236361 22248 raft_consensus.cc:1261] T 0d246b118c1c4388ac04b3775ddbee89 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:54.236929 30875 consensus_queue.cc:1035] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:54.237264 30983 consensus_queue.cc:1035] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:54.242396 30983 raft_consensus.cc:2932] T 0d246b118c1c4388ac04b3775ddbee89 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:54.243156 22248 raft_consensus.cc:2932] T 0d246b118c1c4388ac04b3775ddbee89 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:54.250736 25001 raft_consensus.cc:2932] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:54.253499 21715 catalog_manager.cc:5484] T 0d246b118c1c4388ac04b3775ddbee89 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:54.258467 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d246b118c1c4388ac04b3775ddbee89 with delete_type TABLET_DATA_TOMBSTONED (TS c64840657cdf4b93980bd34120c94d9a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40366
I20240502 01:54:54.260043 31517 tablet_replica.cc:330] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:54:54.260298 31517 raft_consensus.cc:2229] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:54.260591 31517 raft_consensus.cc:2258] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:54.261456 31517 ts_tablet_manager.cc:1875] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:54.267663 31517 ts_tablet_manager.cc:1888] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:54:54.267804 31517 log.cc:1196] T 0d246b118c1c4388ac04b3775ddbee89 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/0d246b118c1c4388ac04b3775ddbee89
I20240502 01:54:54.269049 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 0d246b118c1c4388ac04b3775ddbee89 (table rebalance_test_table_5 [id=868965c86d0d413fbbeaca41748259c6]) successfully deleted
I20240502 01:54:54.762825 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:54.779877 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:54.793807 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:54:54.803962 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:54.819041 24583 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:54.837252 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:54.905822 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:54.922663 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:55.108482 24731 consensus_queue.cc:237] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:55.110801 21986 raft_consensus.cc:1261] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:55.110911 25001 raft_consensus.cc:1261] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:55.111423 31570 consensus_queue.cc:1035] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:55.111701 31423 consensus_queue.cc:1035] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:55.114889 31423 raft_consensus.cc:2932] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:55.117545 25001 raft_consensus.cc:2932] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:55.122521 21711 catalog_manager.cc:5484] T 6dad8caf4f9744eea7be75d0a4890f65 P 28157bc3e678428fb5f93c077d528513 reported cstate change: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:55.129985 21986 raft_consensus.cc:2932] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:55.135416 24583 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dad8caf4f9744eea7be75d0a4890f65 with delete_type TABLET_DATA_TOMBSTONED (TS 0c64d40b5d7644bd8b29feaa80eab72f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53832
I20240502 01:54:55.145776 31618 tablet_replica.cc:330] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:54:55.146127 31618 raft_consensus.cc:2229] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:55.146416 31618 raft_consensus.cc:2258] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:55.147142 31618 ts_tablet_manager.cc:1875] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:55.152992 31618 ts_tablet_manager.cc:1888] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:54:55.153113 31618 log.cc:1196] T 6dad8caf4f9744eea7be75d0a4890f65 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/6dad8caf4f9744eea7be75d0a4890f65
I20240502 01:54:55.153880 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet 6dad8caf4f9744eea7be75d0a4890f65 (table rebalancer_extra_table_18 [id=869549e86fb24c718467c2625a66e8c1]) successfully deleted
W20240502 01:54:55.481096 21792 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.195:38879, user_credentials={real_user=slave}} blocked reactor thread for 70854.2us
W20240502 01:54:55.482206 21790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.194:44411, user_credentials={real_user=slave}} blocked reactor thread for 65574.4us
I20240502 01:54:55.569378 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:54:55.597265 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:55.622578 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:55.631632 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:55.654428 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:55.664180 24583 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:55.738644 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:55.816785 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:56.004542 22251 consensus_queue.cc:237] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:56.006765 21856 raft_consensus.cc:1261] T 9b186753ceeb4a3c9f22391bb9a5b25f P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:56.006966 24867 raft_consensus.cc:1261] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08 [term 2 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:56.007273 31429 consensus_queue.cc:1035] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:56.007565 30918 consensus_queue.cc:1035] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:56.010272 31590 raft_consensus.cc:2932] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:56.010823 24867 raft_consensus.cc:2932] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:56.010860 21856 raft_consensus.cc:2932] T 9b186753ceeb4a3c9f22391bb9a5b25f P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:56.017699 21706 catalog_manager.cc:5484] T 9b186753ceeb4a3c9f22391bb9a5b25f P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: config changed from index 2 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New cstate: current_term: 2 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:56.035976 24715 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b186753ceeb4a3c9f22391bb9a5b25f with delete_type TABLET_DATA_TOMBSTONED (TS 28157bc3e678428fb5f93c077d528513 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41650
I20240502 01:54:56.036587 31739 tablet_replica.cc:330] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:54:56.036808 31739 raft_consensus.cc:2229] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:56.037084 31739 raft_consensus.cc:2258] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:56.037846 31739 ts_tablet_manager.cc:1875] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:56.043912 31739 ts_tablet_manager.cc:1888] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 01:54:56.044027 31739 log.cc:1196] T 9b186753ceeb4a3c9f22391bb9a5b25f P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/9b186753ceeb4a3c9f22391bb9a5b25f
I20240502 01:54:56.044728 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet 9b186753ceeb4a3c9f22391bb9a5b25f (table rebalancer_extra_table_54 [id=aafacdd1649640daab7dbb1308258509]) successfully deleted
I20240502 01:54:56.377979 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:54:56.401659 24583 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:56.402038 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:56.419893 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:56.431298 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:56.497872 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:56.503484 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:56.547794 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:56.681378 24734 consensus_queue.cc:237] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:56.690310 21856 raft_consensus.cc:1261] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:56.690495 24603 raft_consensus.cc:1261] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:56.691169 31608 consensus_queue.cc:1035] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:56.691504 31570 consensus_queue.cc:1035] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:56.694875 31570 raft_consensus.cc:2932] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:56.695483 24603 raft_consensus.cc:2932] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:56.705516 21706 catalog_manager.cc:5484] T 0b73e5695b28424ba127d1f434d1e88c P 28157bc3e678428fb5f93c077d528513 reported cstate change: config changed from index 2 to 4, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New cstate: current_term: 2 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:56.711490 21969 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b73e5695b28424ba127d1f434d1e88c with delete_type TABLET_DATA_TOMBSTONED (TS 06543b05a16f42099bc86db2a0284412 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46720
I20240502 01:54:56.712152 31838 tablet_replica.cc:330] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:54:56.712458 31838 raft_consensus.cc:2229] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:56.715195 31838 raft_consensus.cc:2258] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:56.715327 21856 raft_consensus.cc:2932] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:56.715936 31838 ts_tablet_manager.cc:1875] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:56.723839 31838 ts_tablet_manager.cc:1888] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 01:54:56.724078 31838 log.cc:1196] T 0b73e5695b28424ba127d1f434d1e88c P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/0b73e5695b28424ba127d1f434d1e88c
I20240502 01:54:56.725052 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 0b73e5695b28424ba127d1f434d1e88c (table rebalancer_extra_table_48 [id=ca142102a7e44673b2cac39cee9b5bcd]) successfully deleted
I20240502 01:54:57.067674 24583 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:57.098146 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:57.119712 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:57.121682 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:57.129369 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:57.135591 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:54:57.186100 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 01:54:57.188727 24934 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.199:35737, user_credentials={real_user=slave}} blocked reactor thread for 50092.8us
I20240502 01:54:57.220634 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:57.365669 24599 consensus_queue.cc:237] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:57.367730 21987 raft_consensus.cc:1261] T fa8220d5cb8140e6a66a3b377df56fe4 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:57.368314 25000 raft_consensus.cc:1261] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:57.368325 31558 consensus_queue.cc:1035] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:57.368808 31771 consensus_queue.cc:1035] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:57.371496 31771 raft_consensus.cc:2932] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:57.372071 21987 raft_consensus.cc:2932] T fa8220d5cb8140e6a66a3b377df56fe4 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:57.378159 21708 catalog_manager.cc:5484] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: config changed from index 2 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New cstate: current_term: 2 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:57.380820 25000 raft_consensus.cc:2932] T fa8220d5cb8140e6a66a3b377df56fe4 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:57.425550 24715 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8220d5cb8140e6a66a3b377df56fe4 with delete_type TABLET_DATA_TOMBSTONED (TS 28157bc3e678428fb5f93c077d528513 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41650
I20240502 01:54:57.438037 31947 tablet_replica.cc:330] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:54:57.439920 31947 raft_consensus.cc:2229] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:57.440305 31947 raft_consensus.cc:2258] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:57.441121 31947 ts_tablet_manager.cc:1875] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:57.448248 31947 ts_tablet_manager.cc:1888] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 01:54:57.448374 31947 log.cc:1196] T fa8220d5cb8140e6a66a3b377df56fe4 P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/fa8220d5cb8140e6a66a3b377df56fe4
I20240502 01:54:57.449262 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet fa8220d5cb8140e6a66a3b377df56fe4 (table rebalancer_extra_table_54 [id=aafacdd1649640daab7dbb1308258509]) successfully deleted
I20240502 01:54:57.515450 24734 consensus_queue.cc:237] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:57.518379 24866 raft_consensus.cc:1261] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:57.519059 22251 raft_consensus.cc:1261] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:57.520108 31608 consensus_queue.cc:1035] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:57.520655 31608 consensus_queue.cc:1035] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:54:57.520964 24670 consensus_peers.cc:485] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ad2f9c27d284098a8e4f20ebc4b3d8c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:57.538566 31570 raft_consensus.cc:2932] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:57.541635 22251 raft_consensus.cc:2932] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:57.545429 24867 raft_consensus.cc:2932] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:57.552362 21706 catalog_manager.cc:5484] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:57.602290 21987 consensus_queue.cc:237] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
W20240502 01:54:57.605191 21925 consensus_peers.cc:485] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 -> Peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Couldn't send request to peer 41c70f373c9d49a49891b03a12c6cb2b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d6f1c51e76342429deb0ccbf4ad8ce2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:57.605229 21857 raft_consensus.cc:1261] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:57.605957 30983 consensus_queue.cc:1035] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:57.607818 24867 raft_consensus.cc:1261] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:57.608443 31883 consensus_queue.cc:1035] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:57.621220 31883 raft_consensus.cc:2932] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:57.621941 24867 raft_consensus.cc:2932] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:57.624985 21857 raft_consensus.cc:2932] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:57.627003 21708 catalog_manager.cc:5484] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index -1 to 2, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:57.666505 22251 consensus_queue.cc:237] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:57.669672 22118 raft_consensus.cc:1261] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:57.670352 31590 consensus_queue.cc:1035] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:57.670550 21857 raft_consensus.cc:1261] T 5d40847a56704b33b771c3fd87d6dfae P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:57.671090 22187 consensus_peers.cc:485] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff -> Peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Couldn't send request to peer 41c70f373c9d49a49891b03a12c6cb2b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d40847a56704b33b771c3fd87d6dfae. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:57.671523 31590 consensus_queue.cc:1035] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:57.686239 31839 raft_consensus.cc:2932] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:57.692211 21857 raft_consensus.cc:2932] T 5d40847a56704b33b771c3fd87d6dfae P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:57.694314 21712 catalog_manager.cc:5484] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:57.697022 22118 raft_consensus.cc:2932] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:57.763496 22248 consensus_queue.cc:237] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
W20240502 01:54:57.767709 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d81f9cab671a4258aee1dd4ea4a5ee6b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:57.769013 22118 raft_consensus.cc:1261] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:57.769376 21857 raft_consensus.cc:1261] T d81f9cab671a4258aee1dd4ea4a5ee6b P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:57.769541 31590 consensus_queue.cc:1035] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:57.769889 32009 consensus_queue.cc:1035] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:57.772729 32009 raft_consensus.cc:2932] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:57.773350 21857 raft_consensus.cc:2932] T d81f9cab671a4258aee1dd4ea4a5ee6b P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:57.777685 22118 raft_consensus.cc:2932] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:57.779608 21708 catalog_manager.cc:5484] T d81f9cab671a4258aee1dd4ea4a5ee6b P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:57.835747 21987 consensus_queue.cc:237] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:57.838304 21857 raft_consensus.cc:1261] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:57.838938 24867 raft_consensus.cc:1261] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:57.839001 31989 consensus_queue.cc:1035] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:54:57.838994 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Couldn't send request to peer 0c64d40b5d7644bd8b29feaa80eab72f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f8200b044ef44e1a0451baf2b1485d2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:57.839876 31989 consensus_queue.cc:1035] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:57.842507 30983 raft_consensus.cc:2932] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:57.848639 24867 raft_consensus.cc:2932] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:57.853346 21857 raft_consensus.cc:2932] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:57.856391 21712 catalog_manager.cc:5484] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index 5 to 6, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:54:57.956275 21987 consensus_queue.cc:237] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
W20240502 01:54:57.962549 21924 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9fb0b24337a4175972db268aac5df34. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:57.963523 24734 raft_consensus.cc:1261] T a9fb0b24337a4175972db268aac5df34 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:57.963640 24603 raft_consensus.cc:1261] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:57.964193 31989 consensus_queue.cc:1035] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:57.964768 30983 consensus_queue.cc:1035] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:57.968201 31989 raft_consensus.cc:2932] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:57.968803 24603 raft_consensus.cc:2932] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:57.975378 24734 raft_consensus.cc:2932] T a9fb0b24337a4175972db268aac5df34 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:57.976037 21708 catalog_manager.cc:5484] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:57.995790 32067 ts_tablet_manager.cc:902] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a: Initiating tablet copy from peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:58.001967 32067 tablet_copy_client.cc:308] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.198:42375
I20240502 01:54:58.002715 24755 tablet_copy_service.cc:140] P 28157bc3e678428fb5f93c077d528513: Received BeginTabletCopySession request for tablet 9ad2f9c27d284098a8e4f20ebc4b3d8c from peer c64840657cdf4b93980bd34120c94d9a ({username='slave'} at 127.14.212.195:50687)
I20240502 01:54:58.002872 24755 tablet_copy_service.cc:161] P 28157bc3e678428fb5f93c077d528513: Beginning new tablet copy session on tablet 9ad2f9c27d284098a8e4f20ebc4b3d8c from peer c64840657cdf4b93980bd34120c94d9a at {username='slave'} at 127.14.212.195:50687: session id = c64840657cdf4b93980bd34120c94d9a-9ad2f9c27d284098a8e4f20ebc4b3d8c
I20240502 01:54:58.005555 24755 tablet_copy_source_session.cc:205] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:58.007005 32067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad2f9c27d284098a8e4f20ebc4b3d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:58.012713 32067 tablet_copy_client.cc:783] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:58.013008 32067 tablet_copy_client.cc:647] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:58.016072 32067 tablet_copy_client.cc:523] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:58.019344 32067 tablet_bootstrap.cc:492] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:58.025164 32067 tablet_bootstrap.cc:492] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:58.025523 32067 tablet_bootstrap.cc:492] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a: Bootstrap complete.
I20240502 01:54:58.025732 32067 ts_tablet_manager.cc:1367] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.006s user 0.007s sys 0.000s
I20240502 01:54:58.026399 32067 raft_consensus.cc:348] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:58.026603 32067 raft_consensus.cc:729] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:58.026813 32067 consensus_queue.cc:260] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:58.027348 32067 ts_tablet_manager.cc:1398] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:58.028796 24755 tablet_copy_service.cc:342] P 28157bc3e678428fb5f93c077d528513: Request end of tablet copy session c64840657cdf4b93980bd34120c94d9a-9ad2f9c27d284098a8e4f20ebc4b3d8c received from {username='slave'} at 127.14.212.195:50687
I20240502 01:54:58.028950 24755 tablet_copy_service.cc:433] P 28157bc3e678428fb5f93c077d528513: ending tablet copy session c64840657cdf4b93980bd34120c94d9a-9ad2f9c27d284098a8e4f20ebc4b3d8c on tablet 9ad2f9c27d284098a8e4f20ebc4b3d8c with peer c64840657cdf4b93980bd34120c94d9a
I20240502 01:54:58.030330 25001 consensus_queue.cc:237] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:58.042273 21857 raft_consensus.cc:1261] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:58.042273 24734 raft_consensus.cc:1261] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:58.042626 24936 consensus_peers.cc:485] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Couldn't send request to peer c2a28baa7ee2469294d94b7fe5455cff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e8d8fbd2d1249d5b5de5f189b0d03e9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:58.042984 29819 consensus_queue.cc:1035] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:58.043298 32090 consensus_queue.cc:1035] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:58.046437 29819 raft_consensus.cc:2932] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:58.047410 21856 raft_consensus.cc:2932] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:58.051241 24734 raft_consensus.cc:2932] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:58.062119 21708 catalog_manager.cc:5484] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:58.101125 21987 consensus_queue.cc:237] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:58.104112 25001 raft_consensus.cc:1261] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:58.104223 24734 raft_consensus.cc:1261] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:58.104416 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fafa4ef17fac41b68e3bf4aaa8a4caa8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:58.104802 31989 consensus_queue.cc:1035] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:58.105191 31883 consensus_queue.cc:1035] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:58.115288 30983 raft_consensus.cc:2932] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:58.115711 25001 raft_consensus.cc:2932] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:58.121279 24734 raft_consensus.cc:2932] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:58.126288 21706 catalog_manager.cc:5484] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:58.165354 24734 consensus_queue.cc:237] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:58.168219 24603 raft_consensus.cc:1261] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:58.168247 24867 raft_consensus.cc:1261] T 0a1b190978514cb49efb2b97cc97a65f P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:58.168838 24668 consensus_peers.cc:485] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a1b190978514cb49efb2b97cc97a65f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:58.168905 32130 consensus_queue.cc:1035] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:58.169364 31570 consensus_queue.cc:1035] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:58.172771 32130 raft_consensus.cc:2932] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:58.173439 24603 raft_consensus.cc:2932] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:58.179174 24867 raft_consensus.cc:2932] T 0a1b190978514cb49efb2b97cc97a65f P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:58.179960 21712 catalog_manager.cc:5484] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 reported cstate change: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:54:58.189416 32134 ts_tablet_manager.cc:902] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b: Initiating tablet copy from peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:58.195704 32134 tablet_copy_client.cc:308] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.196:43093
I20240502 01:54:58.196609 22271 tablet_copy_service.cc:140] P c2a28baa7ee2469294d94b7fe5455cff: Received BeginTabletCopySession request for tablet 5d40847a56704b33b771c3fd87d6dfae from peer 41c70f373c9d49a49891b03a12c6cb2b ({username='slave'} at 127.14.212.200:39731)
I20240502 01:54:58.196791 22271 tablet_copy_service.cc:161] P c2a28baa7ee2469294d94b7fe5455cff: Beginning new tablet copy session on tablet 5d40847a56704b33b771c3fd87d6dfae from peer 41c70f373c9d49a49891b03a12c6cb2b at {username='slave'} at 127.14.212.200:39731: session id = 41c70f373c9d49a49891b03a12c6cb2b-5d40847a56704b33b771c3fd87d6dfae
I20240502 01:54:58.200052 22271 tablet_copy_source_session.cc:205] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:58.202090 32134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d40847a56704b33b771c3fd87d6dfae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:58.207835 32134 tablet_copy_client.cc:783] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:58.208045 32134 tablet_copy_client.cc:647] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:58.210628 32134 tablet_copy_client.cc:523] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:58.213938 32134 tablet_bootstrap.cc:492] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:58.227981 32155 ts_tablet_manager.cc:902] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b: Initiating tablet copy from peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:58.229713 24867 consensus_queue.cc:237] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:58.238349 22118 raft_consensus.cc:1261] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:58.240680 24603 raft_consensus.cc:1261] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:58.240707 24801 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c351a8e007d04327bb14303d7b32d5be. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:58.242697 32159 consensus_queue.cc:1035] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:58.243239 30943 consensus_queue.cc:1035] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:58.254657 32160 ts_tablet_manager.cc:902] T d81f9cab671a4258aee1dd4ea4a5ee6b P d7449bb1df214bf4a5f93e4631312f08: Initiating tablet copy from peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:58.254909 30943 raft_consensus.cc:2932] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:58.261548 32155 tablet_copy_client.cc:308] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:44411
I20240502 01:54:58.262416 22010 tablet_copy_service.cc:140] P 06543b05a16f42099bc86db2a0284412: Received BeginTabletCopySession request for tablet 2d6f1c51e76342429deb0ccbf4ad8ce2 from peer 41c70f373c9d49a49891b03a12c6cb2b ({username='slave'} at 127.14.212.200:60949)
I20240502 01:54:58.262631 22010 tablet_copy_service.cc:161] P 06543b05a16f42099bc86db2a0284412: Beginning new tablet copy session on tablet 2d6f1c51e76342429deb0ccbf4ad8ce2 from peer 41c70f373c9d49a49891b03a12c6cb2b at {username='slave'} at 127.14.212.200:60949: session id = 41c70f373c9d49a49891b03a12c6cb2b-2d6f1c51e76342429deb0ccbf4ad8ce2
I20240502 01:54:58.265704 22010 tablet_copy_source_session.cc:205] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:58.267855 32155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6f1c51e76342429deb0ccbf4ad8ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:58.268280 22118 raft_consensus.cc:2932] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:58.276249 32155 tablet_copy_client.cc:783] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:58.276577 32155 tablet_copy_client.cc:647] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:58.277151 21708 catalog_manager.cc:5484] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:58.278316 32155 tablet_copy_client.cc:523] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:58.281692 32155 tablet_bootstrap.cc:492] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:58.294638 32155 tablet_bootstrap.cc:492] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:58.295292 32155 tablet_bootstrap.cc:492] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap complete.
I20240502 01:54:58.295640 32155 ts_tablet_manager.cc:1367] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20240502 01:54:58.296492 32155 raft_consensus.cc:348] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:58.296836 32155 raft_consensus.cc:729] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:58.297202 32155 consensus_queue.cc:260] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:58.297819 32155 ts_tablet_manager.cc:1398] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:58.301195 24603 raft_consensus.cc:2932] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:54:58.303342 22010 tablet_copy_service.cc:342] P 06543b05a16f42099bc86db2a0284412: Request end of tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-2d6f1c51e76342429deb0ccbf4ad8ce2 received from {username='slave'} at 127.14.212.200:60949
I20240502 01:54:58.303480 22010 tablet_copy_service.cc:433] P 06543b05a16f42099bc86db2a0284412: ending tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-2d6f1c51e76342429deb0ccbf4ad8ce2 on tablet 2d6f1c51e76342429deb0ccbf4ad8ce2 with peer 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:58.317560 32134 tablet_bootstrap.cc:492] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:58.318068 32134 tablet_bootstrap.cc:492] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap complete.
I20240502 01:54:58.318403 32134 ts_tablet_manager.cc:1367] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.105s user 0.040s sys 0.012s
I20240502 01:54:58.319293 32134 raft_consensus.cc:348] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:58.319597 32134 raft_consensus.cc:729] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:58.319864 32134 consensus_queue.cc:260] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:58.320569 32134 ts_tablet_manager.cc:1398] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:54:58.321596 22271 tablet_copy_service.cc:342] P c2a28baa7ee2469294d94b7fe5455cff: Request end of tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-5d40847a56704b33b771c3fd87d6dfae received from {username='slave'} at 127.14.212.200:39731
I20240502 01:54:58.321717 22271 tablet_copy_service.cc:433] P c2a28baa7ee2469294d94b7fe5455cff: ending tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-5d40847a56704b33b771c3fd87d6dfae on tablet 5d40847a56704b33b771c3fd87d6dfae with peer 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:58.344957 32160 tablet_copy_client.cc:308] T d81f9cab671a4258aee1dd4ea4a5ee6b P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.196:43093
I20240502 01:54:58.345906 22271 tablet_copy_service.cc:140] P c2a28baa7ee2469294d94b7fe5455cff: Received BeginTabletCopySession request for tablet d81f9cab671a4258aee1dd4ea4a5ee6b from peer d7449bb1df214bf4a5f93e4631312f08 ({username='slave'} at 127.14.212.199:55493)
I20240502 01:54:58.346056 22271 tablet_copy_service.cc:161] P c2a28baa7ee2469294d94b7fe5455cff: Beginning new tablet copy session on tablet d81f9cab671a4258aee1dd4ea4a5ee6b from peer d7449bb1df214bf4a5f93e4631312f08 at {username='slave'} at 127.14.212.199:55493: session id = d7449bb1df214bf4a5f93e4631312f08-d81f9cab671a4258aee1dd4ea4a5ee6b
I20240502 01:54:58.349009 22271 tablet_copy_source_session.cc:205] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:58.350942 32160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81f9cab671a4258aee1dd4ea4a5ee6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:58.358240 32160 tablet_copy_client.cc:783] T d81f9cab671a4258aee1dd4ea4a5ee6b P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:58.358471 32160 tablet_copy_client.cc:647] T d81f9cab671a4258aee1dd4ea4a5ee6b P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:58.360744 32160 tablet_copy_client.cc:523] T d81f9cab671a4258aee1dd4ea4a5ee6b P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:58.364477 32160 tablet_bootstrap.cc:492] T d81f9cab671a4258aee1dd4ea4a5ee6b P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:58.414407 32182 ts_tablet_manager.cc:902] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f: Initiating tablet copy from peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:58.416276 32067 ts_tablet_manager.cc:902] T a9fb0b24337a4175972db268aac5df34 P c64840657cdf4b93980bd34120c94d9a: Initiating tablet copy from peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:58.435145 32182 tablet_copy_client.cc:308] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:44411
I20240502 01:54:58.436062 22010 tablet_copy_service.cc:140] P 06543b05a16f42099bc86db2a0284412: Received BeginTabletCopySession request for tablet 0f8200b044ef44e1a0451baf2b1485d2 from peer 0c64d40b5d7644bd8b29feaa80eab72f ({username='slave'} at 127.14.212.197:35233)
I20240502 01:54:58.436286 22010 tablet_copy_service.cc:161] P 06543b05a16f42099bc86db2a0284412: Beginning new tablet copy session on tablet 0f8200b044ef44e1a0451baf2b1485d2 from peer 0c64d40b5d7644bd8b29feaa80eab72f at {username='slave'} at 127.14.212.197:35233: session id = 0c64d40b5d7644bd8b29feaa80eab72f-0f8200b044ef44e1a0451baf2b1485d2
I20240502 01:54:58.440276 22010 tablet_copy_source_session.cc:205] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:58.443861 32182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8200b044ef44e1a0451baf2b1485d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:58.458699 32182 tablet_copy_client.cc:783] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:58.458977 32182 tablet_copy_client.cc:647] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:58.467233 32182 tablet_copy_client.cc:523] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:58.471050 32182 tablet_bootstrap.cc:492] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:54:58.471855 24864 consensus_queue.cc:237] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:58.475340 32160 tablet_bootstrap.cc:492] T d81f9cab671a4258aee1dd4ea4a5ee6b P d7449bb1df214bf4a5f93e4631312f08: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:58.475785 32160 tablet_bootstrap.cc:492] T d81f9cab671a4258aee1dd4ea4a5ee6b P d7449bb1df214bf4a5f93e4631312f08: Bootstrap complete.
I20240502 01:54:58.476038 32160 ts_tablet_manager.cc:1367] T d81f9cab671a4258aee1dd4ea4a5ee6b P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.112s user 0.059s sys 0.000s
I20240502 01:54:58.476807 32160 raft_consensus.cc:348] T d81f9cab671a4258aee1dd4ea4a5ee6b P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:58.477035 32160 raft_consensus.cc:729] T d81f9cab671a4258aee1dd4ea4a5ee6b P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:58.477349 24603 raft_consensus.cc:1261] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:58.477272 32160 consensus_queue.cc:260] T d81f9cab671a4258aee1dd4ea4a5ee6b P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
W20240502 01:54:58.477393 24803 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 -> Peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Couldn't send request to peer c2a28baa7ee2469294d94b7fe5455cff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3237002dce97460799e219cf5b4968cb. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:58.477826 32160 ts_tablet_manager.cc:1398] T d81f9cab671a4258aee1dd4ea4a5ee6b P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:54:58.478155 24734 raft_consensus.cc:1261] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:58.478267 31903 consensus_queue.cc:1035] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:58.478643 22271 tablet_copy_service.cc:342] P c2a28baa7ee2469294d94b7fe5455cff: Request end of tablet copy session d7449bb1df214bf4a5f93e4631312f08-d81f9cab671a4258aee1dd4ea4a5ee6b received from {username='slave'} at 127.14.212.199:55493
I20240502 01:54:58.478760 22271 tablet_copy_service.cc:433] P c2a28baa7ee2469294d94b7fe5455cff: ending tablet copy session d7449bb1df214bf4a5f93e4631312f08-d81f9cab671a4258aee1dd4ea4a5ee6b on tablet d81f9cab671a4258aee1dd4ea4a5ee6b with peer d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:58.479650 31903 consensus_queue.cc:1035] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:58.497918 32067 tablet_copy_client.cc:308] T a9fb0b24337a4175972db268aac5df34 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:44411
I20240502 01:54:58.498730 22010 tablet_copy_service.cc:140] P 06543b05a16f42099bc86db2a0284412: Received BeginTabletCopySession request for tablet a9fb0b24337a4175972db268aac5df34 from peer c64840657cdf4b93980bd34120c94d9a ({username='slave'} at 127.14.212.195:42095)
I20240502 01:54:58.498915 22010 tablet_copy_service.cc:161] P 06543b05a16f42099bc86db2a0284412: Beginning new tablet copy session on tablet a9fb0b24337a4175972db268aac5df34 from peer c64840657cdf4b93980bd34120c94d9a at {username='slave'} at 127.14.212.195:42095: session id = c64840657cdf4b93980bd34120c94d9a-a9fb0b24337a4175972db268aac5df34
I20240502 01:54:58.501828 22010 tablet_copy_source_session.cc:205] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:58.504053 32067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fb0b24337a4175972db268aac5df34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:58.509464 30943 raft_consensus.cc:2932] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:58.512374 32067 tablet_copy_client.cc:783] T a9fb0b24337a4175972db268aac5df34 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:58.512681 32067 tablet_copy_client.cc:647] T a9fb0b24337a4175972db268aac5df34 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:58.513972 24734 raft_consensus.cc:2932] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:58.515638 32067 tablet_copy_client.cc:523] T a9fb0b24337a4175972db268aac5df34 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:58.520718 32067 tablet_bootstrap.cc:492] T a9fb0b24337a4175972db268aac5df34 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:58.523283 24603 raft_consensus.cc:2932] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:58.532788 21708 catalog_manager.cc:5484] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:58.533520 32067 tablet_bootstrap.cc:492] T a9fb0b24337a4175972db268aac5df34 P c64840657cdf4b93980bd34120c94d9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:58.534039 32067 tablet_bootstrap.cc:492] T a9fb0b24337a4175972db268aac5df34 P c64840657cdf4b93980bd34120c94d9a: Bootstrap complete.
I20240502 01:54:58.534454 32067 ts_tablet_manager.cc:1367] T a9fb0b24337a4175972db268aac5df34 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.001s
I20240502 01:54:58.535462 32067 raft_consensus.cc:348] T a9fb0b24337a4175972db268aac5df34 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:58.535809 32067 raft_consensus.cc:729] T a9fb0b24337a4175972db268aac5df34 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:58.536082 32067 consensus_queue.cc:260] T a9fb0b24337a4175972db268aac5df34 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:58.538789 32067 ts_tablet_manager.cc:1398] T a9fb0b24337a4175972db268aac5df34 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:54:58.539552 22010 tablet_copy_service.cc:342] P 06543b05a16f42099bc86db2a0284412: Request end of tablet copy session c64840657cdf4b93980bd34120c94d9a-a9fb0b24337a4175972db268aac5df34 received from {username='slave'} at 127.14.212.195:42095
I20240502 01:54:58.539701 22010 tablet_copy_service.cc:433] P 06543b05a16f42099bc86db2a0284412: ending tablet copy session c64840657cdf4b93980bd34120c94d9a-a9fb0b24337a4175972db268aac5df34 on tablet a9fb0b24337a4175972db268aac5df34 with peer c64840657cdf4b93980bd34120c94d9a
I20240502 01:54:58.555294 32182 tablet_bootstrap.cc:492] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:58.555764 32182 tablet_bootstrap.cc:492] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap complete.
I20240502 01:54:58.556077 32182 ts_tablet_manager.cc:1367] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.085s user 0.044s sys 0.016s
I20240502 01:54:58.556671 32182 raft_consensus.cc:348] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:58.556852 32182 raft_consensus.cc:729] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:54:58.557144 32182 consensus_queue.cc:260] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:54:58.557703 32182 ts_tablet_manager.cc:1398] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:54:58.558470 22010 tablet_copy_service.cc:342] P 06543b05a16f42099bc86db2a0284412: Request end of tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-0f8200b044ef44e1a0451baf2b1485d2 received from {username='slave'} at 127.14.212.197:35233
I20240502 01:54:58.558612 22010 tablet_copy_service.cc:433] P 06543b05a16f42099bc86db2a0284412: ending tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-0f8200b044ef44e1a0451baf2b1485d2 on tablet 0f8200b044ef44e1a0451baf2b1485d2 with peer 0c64d40b5d7644bd8b29feaa80eab72f
I20240502 01:54:58.594532 22118 raft_consensus.cc:1203] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:58.604527 32216 ts_tablet_manager.cc:902] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P c2a28baa7ee2469294d94b7fe5455cff: Initiating tablet copy from peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:54:58.606768 32160 ts_tablet_manager.cc:902] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P d7449bb1df214bf4a5f93e4631312f08: Initiating tablet copy from peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:54:58.616338 21856 consensus_queue.cc:237] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:58.624050 32216 tablet_copy_client.cc:308] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.200:37097
W20240502 01:54:58.624955 21793 consensus_peers.cc:485] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 -> Peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Couldn't send request to peer 41c70f373c9d49a49891b03a12c6cb2b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea103b58ddcc4459af3d51878a110d38. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:58.624926 25021 tablet_copy_service.cc:140] P 41c70f373c9d49a49891b03a12c6cb2b: Received BeginTabletCopySession request for tablet 6e8d8fbd2d1249d5b5de5f189b0d03e9 from peer c2a28baa7ee2469294d94b7fe5455cff ({username='slave'} at 127.14.212.196:54787)
I20240502 01:54:58.625135 25021 tablet_copy_service.cc:161] P 41c70f373c9d49a49891b03a12c6cb2b: Beginning new tablet copy session on tablet 6e8d8fbd2d1249d5b5de5f189b0d03e9 from peer c2a28baa7ee2469294d94b7fe5455cff at {username='slave'} at 127.14.212.196:54787: session id = c2a28baa7ee2469294d94b7fe5455cff-6e8d8fbd2d1249d5b5de5f189b0d03e9
I20240502 01:54:58.625783 24867 raft_consensus.cc:1203] T d81f9cab671a4258aee1dd4ea4a5ee6b P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:58.629312 25021 tablet_copy_source_session.cc:205] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:58.630826 24603 raft_consensus.cc:1261] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:58.631593 31311 consensus_queue.cc:1035] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:58.635454 32216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e8d8fbd2d1249d5b5de5f189b0d03e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:58.638221 25001 raft_consensus.cc:1203] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:58.641513 24867 raft_consensus.cc:1261] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:58.643698 31690 consensus_queue.cc:1035] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:58.654958 32216 tablet_copy_client.cc:783] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:58.655673 32216 tablet_copy_client.cc:647] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:58.657938 32216 tablet_copy_client.cc:523] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:58.668000 32216 tablet_bootstrap.cc:492] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:58.687933 32160 tablet_copy_client.cc:308] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:44411
I20240502 01:54:58.688791 22010 tablet_copy_service.cc:140] P 06543b05a16f42099bc86db2a0284412: Received BeginTabletCopySession request for tablet fafa4ef17fac41b68e3bf4aaa8a4caa8 from peer d7449bb1df214bf4a5f93e4631312f08 ({username='slave'} at 127.14.212.199:39887)
I20240502 01:54:58.689062 22010 tablet_copy_service.cc:161] P 06543b05a16f42099bc86db2a0284412: Beginning new tablet copy session on tablet fafa4ef17fac41b68e3bf4aaa8a4caa8 from peer d7449bb1df214bf4a5f93e4631312f08 at {username='slave'} at 127.14.212.199:39887: session id = d7449bb1df214bf4a5f93e4631312f08-fafa4ef17fac41b68e3bf4aaa8a4caa8
I20240502 01:54:58.693468 22010 tablet_copy_source_session.cc:205] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:58.695195 32160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafa4ef17fac41b68e3bf4aaa8a4caa8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:54:58.695863 21791 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.199:35737, user_credentials={real_user=slave}} blocked reactor thread for 52981.4us
I20240502 01:54:58.703179 32216 tablet_bootstrap.cc:492] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:58.703495 32160 tablet_copy_client.cc:783] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:58.703575 32216 tablet_bootstrap.cc:492] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap complete.
I20240502 01:54:58.703747 32160 tablet_copy_client.cc:647] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:58.703848 32216 ts_tablet_manager.cc:1367] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.002s
I20240502 01:54:58.704586 32216 raft_consensus.cc:348] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:58.705297 32216 raft_consensus.cc:729] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:58.705572 32216 consensus_queue.cc:260] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:58.706341 32216 ts_tablet_manager.cc:1398] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20240502 01:54:58.707540 25021 tablet_copy_service.cc:342] P 41c70f373c9d49a49891b03a12c6cb2b: Request end of tablet copy session c2a28baa7ee2469294d94b7fe5455cff-6e8d8fbd2d1249d5b5de5f189b0d03e9 received from {username='slave'} at 127.14.212.196:54787
I20240502 01:54:58.707692 25021 tablet_copy_service.cc:433] P 41c70f373c9d49a49891b03a12c6cb2b: ending tablet copy session c2a28baa7ee2469294d94b7fe5455cff-6e8d8fbd2d1249d5b5de5f189b0d03e9 on tablet 6e8d8fbd2d1249d5b5de5f189b0d03e9 with peer c2a28baa7ee2469294d94b7fe5455cff
I20240502 01:54:58.712821 32160 tablet_copy_client.cc:523] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:58.713223 31311 raft_consensus.cc:2932] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:58.717550 32160 tablet_bootstrap.cc:492] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:58.722165 24864 raft_consensus.cc:2932] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:58.731745 21706 catalog_manager.cc:5484] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: config changed from index -1 to 2, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:58.736085 24603 raft_consensus.cc:2932] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:54:58.743284 32160 tablet_bootstrap.cc:492] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:58.743690 32160 tablet_bootstrap.cc:492] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap complete.
I20240502 01:54:58.743911 32160 ts_tablet_manager.cc:1367] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.001s
I20240502 01:54:58.744594 32160 raft_consensus.cc:348] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:58.744827 32160 raft_consensus.cc:729] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:58.745075 32160 consensus_queue.cc:260] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:58.745608 32160 ts_tablet_manager.cc:1398] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:54:58.747555 22010 tablet_copy_service.cc:342] P 06543b05a16f42099bc86db2a0284412: Request end of tablet copy session d7449bb1df214bf4a5f93e4631312f08-fafa4ef17fac41b68e3bf4aaa8a4caa8 received from {username='slave'} at 127.14.212.199:39887
I20240502 01:54:58.747720 22010 tablet_copy_service.cc:433] P 06543b05a16f42099bc86db2a0284412: ending tablet copy session d7449bb1df214bf4a5f93e4631312f08-fafa4ef17fac41b68e3bf4aaa8a4caa8 on tablet fafa4ef17fac41b68e3bf4aaa8a4caa8 with peer d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:58.760866 32236 ts_tablet_manager.cc:902] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412: Initiating tablet copy from peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:58.766270 32236 tablet_copy_client.cc:308] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.199:35737
I20240502 01:54:58.767264 24888 tablet_copy_service.cc:140] P d7449bb1df214bf4a5f93e4631312f08: Received BeginTabletCopySession request for tablet c351a8e007d04327bb14303d7b32d5be from peer 06543b05a16f42099bc86db2a0284412 ({username='slave'} at 127.14.212.194:53127)
I20240502 01:54:58.767421 24888 tablet_copy_service.cc:161] P d7449bb1df214bf4a5f93e4631312f08: Beginning new tablet copy session on tablet c351a8e007d04327bb14303d7b32d5be from peer 06543b05a16f42099bc86db2a0284412 at {username='slave'} at 127.14.212.194:53127: session id = 06543b05a16f42099bc86db2a0284412-c351a8e007d04327bb14303d7b32d5be
I20240502 01:54:58.770215 24888 tablet_copy_source_session.cc:205] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:58.771790 32236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c351a8e007d04327bb14303d7b32d5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:58.777707 32236 tablet_copy_client.cc:783] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:58.778033 32236 tablet_copy_client.cc:647] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:58.780850 32236 tablet_copy_client.cc:523] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:58.785977 32250 ts_tablet_manager.cc:902] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77: Initiating tablet copy from peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:54:58.787349 32236 tablet_bootstrap.cc:492] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:54:58.792488 32250 tablet_copy_client.cc:308] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.198:42375
I20240502 01:54:58.793989 24755 tablet_copy_service.cc:140] P 28157bc3e678428fb5f93c077d528513: Received BeginTabletCopySession request for tablet 0a1b190978514cb49efb2b97cc97a65f from peer 759257ea7bef415ca698e60d82199d77 ({username='slave'} at 127.14.212.193:45457)
I20240502 01:54:58.794219 24755 tablet_copy_service.cc:161] P 28157bc3e678428fb5f93c077d528513: Beginning new tablet copy session on tablet 0a1b190978514cb49efb2b97cc97a65f from peer 759257ea7bef415ca698e60d82199d77 at {username='slave'} at 127.14.212.193:45457: session id = 759257ea7bef415ca698e60d82199d77-0a1b190978514cb49efb2b97cc97a65f
I20240502 01:54:58.795135 32236 tablet_bootstrap.cc:492] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:58.795466 32236 tablet_bootstrap.cc:492] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412: Bootstrap complete.
I20240502 01:54:58.795686 32236 ts_tablet_manager.cc:1367] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240502 01:54:58.796411 32236 raft_consensus.cc:348] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:58.796592 32236 raft_consensus.cc:729] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:54:58.796810 32236 consensus_queue.cc:260] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:54:58.797685 32236 ts_tablet_manager.cc:1398] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:54:58.799813 24888 tablet_copy_service.cc:342] P d7449bb1df214bf4a5f93e4631312f08: Request end of tablet copy session 06543b05a16f42099bc86db2a0284412-c351a8e007d04327bb14303d7b32d5be received from {username='slave'} at 127.14.212.194:53127
I20240502 01:54:58.799950 24888 tablet_copy_service.cc:433] P d7449bb1df214bf4a5f93e4631312f08: ending tablet copy session 06543b05a16f42099bc86db2a0284412-c351a8e007d04327bb14303d7b32d5be on tablet c351a8e007d04327bb14303d7b32d5be with peer 06543b05a16f42099bc86db2a0284412
I20240502 01:54:58.800653 24755 tablet_copy_source_session.cc:205] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:58.802366 32250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a1b190978514cb49efb2b97cc97a65f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:58.809217 32250 tablet_copy_client.cc:783] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:58.809471 32250 tablet_copy_client.cc:647] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:58.811388 32250 tablet_copy_client.cc:523] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:58.815944 32250 tablet_bootstrap.cc:492] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:54:58.816093 25001 raft_consensus.cc:1203] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:58.829460 32250 tablet_bootstrap.cc:492] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:58.829943 32250 tablet_bootstrap.cc:492] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77: Bootstrap complete.
I20240502 01:54:58.830421 32250 ts_tablet_manager.cc:1367] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20240502 01:54:58.831656 32250 raft_consensus.cc:348] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:58.831884 32250 raft_consensus.cc:729] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:54:58.832480 32250 consensus_queue.cc:260] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:54:58.834997 32250 ts_tablet_manager.cc:1398] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:58.836762 24755 tablet_copy_service.cc:342] P 28157bc3e678428fb5f93c077d528513: Request end of tablet copy session 759257ea7bef415ca698e60d82199d77-0a1b190978514cb49efb2b97cc97a65f received from {username='slave'} at 127.14.212.193:45457
I20240502 01:54:58.836908 24755 tablet_copy_service.cc:433] P 28157bc3e678428fb5f93c077d528513: ending tablet copy session 759257ea7bef415ca698e60d82199d77-0a1b190978514cb49efb2b97cc97a65f on tablet 0a1b190978514cb49efb2b97cc97a65f with peer 759257ea7bef415ca698e60d82199d77
I20240502 01:54:58.839977 24864 consensus_queue.cc:237] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:58.843468 22248 raft_consensus.cc:1261] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:58.844331 24803 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ce01a3c9d9948ba8f153ad150cf3cb0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:58.844631 31903 consensus_queue.cc:1035] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:58.845054 24603 raft_consensus.cc:1261] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:58.845844 31903 consensus_queue.cc:1035] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:58.879300 31903 raft_consensus.cc:2932] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:58.888257 24603 raft_consensus.cc:2932] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:58.891477 21708 catalog_manager.cc:5484] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:58.893136 22248 raft_consensus.cc:2932] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:54:58.942770 22118 raft_consensus.cc:1203] T a9fb0b24337a4175972db268aac5df34 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:58.985671 24864 consensus_queue.cc:237] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:58.988368 22118 raft_consensus.cc:1261] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:58.988497 21856 raft_consensus.cc:1261] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:58.988620 24802 consensus_peers.cc:485] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a90f08d29a49d3b451f85cbdc828a6. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:58.989020 32159 consensus_queue.cc:1035] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:58.989289 31903 consensus_queue.cc:1035] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:58.995121 31903 raft_consensus.cc:2932] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:58.995698 21856 raft_consensus.cc:2932] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:58.998752 22118 raft_consensus.cc:2932] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:59.001714 21708 catalog_manager.cc:5484] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:54:59.006382 24603 raft_consensus.cc:1203] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 01:54:59.022073 32216 ts_tablet_manager.cc:902] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff: Initiating tablet copy from peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:59.034324 24864 raft_consensus.cc:1203] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:59.038419 32216 tablet_copy_client.cc:308] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.199:35737
I20240502 01:54:59.040570 24888 tablet_copy_service.cc:140] P d7449bb1df214bf4a5f93e4631312f08: Received BeginTabletCopySession request for tablet 3237002dce97460799e219cf5b4968cb from peer c2a28baa7ee2469294d94b7fe5455cff ({username='slave'} at 127.14.212.196:52697)
I20240502 01:54:59.040727 24888 tablet_copy_service.cc:161] P d7449bb1df214bf4a5f93e4631312f08: Beginning new tablet copy session on tablet 3237002dce97460799e219cf5b4968cb from peer c2a28baa7ee2469294d94b7fe5455cff at {username='slave'} at 127.14.212.196:52697: session id = c2a28baa7ee2469294d94b7fe5455cff-3237002dce97460799e219cf5b4968cb
I20240502 01:54:59.042876 24888 tablet_copy_source_session.cc:205] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:59.044328 32216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3237002dce97460799e219cf5b4968cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:59.050983 32216 tablet_copy_client.cc:783] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:59.051235 32216 tablet_copy_client.cc:647] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:59.053474 32216 tablet_copy_client.cc:523] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:59.057384 32216 tablet_bootstrap.cc:492] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:59.063484 32216 tablet_bootstrap.cc:492] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:59.063881 32216 tablet_bootstrap.cc:492] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap complete.
I20240502 01:54:59.064168 32216 ts_tablet_manager.cc:1367] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 01:54:59.064805 32216 raft_consensus.cc:348] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:59.065040 32216 raft_consensus.cc:729] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:59.065327 32216 consensus_queue.cc:260] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:59.065866 32216 ts_tablet_manager.cc:1398] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:54:59.067756 24888 tablet_copy_service.cc:342] P d7449bb1df214bf4a5f93e4631312f08: Request end of tablet copy session c2a28baa7ee2469294d94b7fe5455cff-3237002dce97460799e219cf5b4968cb received from {username='slave'} at 127.14.212.196:52697
I20240502 01:54:59.067909 24888 tablet_copy_service.cc:433] P d7449bb1df214bf4a5f93e4631312f08: ending tablet copy session c2a28baa7ee2469294d94b7fe5455cff-3237002dce97460799e219cf5b4968cb on tablet 3237002dce97460799e219cf5b4968cb with peer c2a28baa7ee2469294d94b7fe5455cff
I20240502 01:54:59.083359 22248 consensus_queue.cc:237] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:59.086236 21986 raft_consensus.cc:1261] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:59.086212 22118 raft_consensus.cc:1261] T ba39439749a9486799373e173110f5c5 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:59.086846 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba39439749a9486799373e173110f5c5. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:59.086977 31839 consensus_queue.cc:1035] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:59.090342 31839 consensus_queue.cc:1035] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:59.096907 31839 raft_consensus.cc:2932] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:59.097908 21986 raft_consensus.cc:2932] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:59.106422 22118 raft_consensus.cc:2932] T ba39439749a9486799373e173110f5c5 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:59.110810 21706 catalog_manager.cc:5484] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:54:59.148994 21857 consensus_queue.cc:237] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:59.151320 22118 raft_consensus.cc:1261] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:59.151541 25000 raft_consensus.cc:1261] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:59.151979 32233 consensus_queue.cc:1035] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:59.152308 21792 consensus_peers.cc:485] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 -> Peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Couldn't send request to peer c2a28baa7ee2469294d94b7fe5455cff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e00feee2662a48d8a1a57f9e52275154. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:59.152549 32233 consensus_queue.cc:1035] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:59.158046 32233 raft_consensus.cc:2932] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:59.158797 22118 raft_consensus.cc:2932] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:59.165709 21712 catalog_manager.cc:5484] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index 3 to 4, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:59.167097 25000 raft_consensus.cc:2932] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:54:59.175832 21857 raft_consensus.cc:1203] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:59.191447 22248 raft_consensus.cc:1203] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:59.204017 32338 ts_tablet_manager.cc:902] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b: Initiating tablet copy from peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:59.212127 32338 tablet_copy_client.cc:308] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.193:33785
I20240502 01:54:59.212901 21878 tablet_copy_service.cc:140] P 759257ea7bef415ca698e60d82199d77: Received BeginTabletCopySession request for tablet ea103b58ddcc4459af3d51878a110d38 from peer 41c70f373c9d49a49891b03a12c6cb2b ({username='slave'} at 127.14.212.200:36811)
I20240502 01:54:59.213084 21878 tablet_copy_service.cc:161] P 759257ea7bef415ca698e60d82199d77: Beginning new tablet copy session on tablet ea103b58ddcc4459af3d51878a110d38 from peer 41c70f373c9d49a49891b03a12c6cb2b at {username='slave'} at 127.14.212.200:36811: session id = 41c70f373c9d49a49891b03a12c6cb2b-ea103b58ddcc4459af3d51878a110d38
I20240502 01:54:59.217455 21878 tablet_copy_source_session.cc:205] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:59.219216 32338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea103b58ddcc4459af3d51878a110d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:59.228123 32338 tablet_copy_client.cc:783] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:59.228466 32338 tablet_copy_client.cc:647] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:59.230790 32338 tablet_copy_client.cc:523] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:59.234751 32338 tablet_bootstrap.cc:492] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:54:59.241922 32338 tablet_bootstrap.cc:492] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:59.242703 32338 tablet_bootstrap.cc:492] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap complete.
I20240502 01:54:59.242973 32338 ts_tablet_manager.cc:1367] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.006s
I20240502 01:54:59.243696 32338 raft_consensus.cc:348] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:59.243937 32338 raft_consensus.cc:729] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:54:59.244199 32338 consensus_queue.cc:260] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:54:59.244839 32338 ts_tablet_manager.cc:1398] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:59.247639 21878 tablet_copy_service.cc:342] P 759257ea7bef415ca698e60d82199d77: Request end of tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-ea103b58ddcc4459af3d51878a110d38 received from {username='slave'} at 127.14.212.200:36811
I20240502 01:54:59.247777 21878 tablet_copy_service.cc:433] P 759257ea7bef415ca698e60d82199d77: ending tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-ea103b58ddcc4459af3d51878a110d38 on tablet ea103b58ddcc4459af3d51878a110d38 with peer 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:59.299022 21986 raft_consensus.cc:1203] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:59.394026 32368 ts_tablet_manager.cc:902] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a: Initiating tablet copy from peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:59.395759 32368 tablet_copy_client.cc:308] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.199:35737
I20240502 01:54:59.396607 24888 tablet_copy_service.cc:140] P d7449bb1df214bf4a5f93e4631312f08: Received BeginTabletCopySession request for tablet 7ce01a3c9d9948ba8f153ad150cf3cb0 from peer c64840657cdf4b93980bd34120c94d9a ({username='slave'} at 127.14.212.195:37371)
I20240502 01:54:59.396777 24888 tablet_copy_service.cc:161] P d7449bb1df214bf4a5f93e4631312f08: Beginning new tablet copy session on tablet 7ce01a3c9d9948ba8f153ad150cf3cb0 from peer c64840657cdf4b93980bd34120c94d9a at {username='slave'} at 127.14.212.195:37371: session id = c64840657cdf4b93980bd34120c94d9a-7ce01a3c9d9948ba8f153ad150cf3cb0
I20240502 01:54:59.399909 24888 tablet_copy_source_session.cc:205] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:59.429468 32368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce01a3c9d9948ba8f153ad150cf3cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:59.440462 32368 tablet_copy_client.cc:783] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:59.440840 32368 tablet_copy_client.cc:647] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:59.450469 32368 tablet_copy_client.cc:523] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:59.459918 32368 tablet_bootstrap.cc:492] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:54:59.499125 32368 tablet_bootstrap.cc:492] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:59.499598 32368 tablet_bootstrap.cc:492] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a: Bootstrap complete.
I20240502 01:54:59.499853 32368 ts_tablet_manager.cc:1367] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.000s
I20240502 01:54:59.500634 32368 raft_consensus.cc:348] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:59.504128 32368 raft_consensus.cc:729] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:54:59.504439 32368 consensus_queue.cc:260] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:54:59.506737 32368 ts_tablet_manager.cc:1398] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240502 01:54:59.513422 24888 tablet_copy_service.cc:342] P d7449bb1df214bf4a5f93e4631312f08: Request end of tablet copy session c64840657cdf4b93980bd34120c94d9a-7ce01a3c9d9948ba8f153ad150cf3cb0 received from {username='slave'} at 127.14.212.195:37371
I20240502 01:54:59.513588 24888 tablet_copy_service.cc:433] P d7449bb1df214bf4a5f93e4631312f08: ending tablet copy session c64840657cdf4b93980bd34120c94d9a-7ce01a3c9d9948ba8f153ad150cf3cb0 on tablet 7ce01a3c9d9948ba8f153ad150cf3cb0 with peer c64840657cdf4b93980bd34120c94d9a
I20240502 01:54:59.524415 32383 ts_tablet_manager.cc:902] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513: Initiating tablet copy from peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:54:59.526432 32382 ts_tablet_manager.cc:902] T ba39439749a9486799373e173110f5c5 P d7449bb1df214bf4a5f93e4631312f08: Initiating tablet copy from peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:54:59.541132 22248 raft_consensus.cc:1203] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:59.551559 32216 ts_tablet_manager.cc:902] T e00feee2662a48d8a1a57f9e52275154 P c2a28baa7ee2469294d94b7fe5455cff: Initiating tablet copy from peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:54:59.563920 32383 tablet_copy_client.cc:308] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.199:35737
I20240502 01:54:59.565572 24888 tablet_copy_service.cc:140] P d7449bb1df214bf4a5f93e4631312f08: Received BeginTabletCopySession request for tablet 77a90f08d29a49d3b451f85cbdc828a6 from peer 28157bc3e678428fb5f93c077d528513 ({username='slave'} at 127.14.212.198:33761)
I20240502 01:54:59.565797 24888 tablet_copy_service.cc:161] P d7449bb1df214bf4a5f93e4631312f08: Beginning new tablet copy session on tablet 77a90f08d29a49d3b451f85cbdc828a6 from peer 28157bc3e678428fb5f93c077d528513 at {username='slave'} at 127.14.212.198:33761: session id = 28157bc3e678428fb5f93c077d528513-77a90f08d29a49d3b451f85cbdc828a6
I20240502 01:54:59.570016 24888 tablet_copy_source_session.cc:205] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:59.571885 32383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a90f08d29a49d3b451f85cbdc828a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:59.587533 32383 tablet_copy_client.cc:783] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:59.587952 32383 tablet_copy_client.cc:647] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:59.589784 32383 tablet_copy_client.cc:523] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:59.594461 32383 tablet_bootstrap.cc:492] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:54:59.604730 32382 tablet_copy_client.cc:308] T ba39439749a9486799373e173110f5c5 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.196:43093
I20240502 01:54:59.605033 32216 tablet_copy_client.cc:308] T e00feee2662a48d8a1a57f9e52275154 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.193:33785
I20240502 01:54:59.605734 22271 tablet_copy_service.cc:140] P c2a28baa7ee2469294d94b7fe5455cff: Received BeginTabletCopySession request for tablet ba39439749a9486799373e173110f5c5 from peer d7449bb1df214bf4a5f93e4631312f08 ({username='slave'} at 127.14.212.199:55493)
I20240502 01:54:59.605834 21878 tablet_copy_service.cc:140] P 759257ea7bef415ca698e60d82199d77: Received BeginTabletCopySession request for tablet e00feee2662a48d8a1a57f9e52275154 from peer c2a28baa7ee2469294d94b7fe5455cff ({username='slave'} at 127.14.212.196:45367)
I20240502 01:54:59.605929 22271 tablet_copy_service.cc:161] P c2a28baa7ee2469294d94b7fe5455cff: Beginning new tablet copy session on tablet ba39439749a9486799373e173110f5c5 from peer d7449bb1df214bf4a5f93e4631312f08 at {username='slave'} at 127.14.212.199:55493: session id = d7449bb1df214bf4a5f93e4631312f08-ba39439749a9486799373e173110f5c5
I20240502 01:54:59.605981 21878 tablet_copy_service.cc:161] P 759257ea7bef415ca698e60d82199d77: Beginning new tablet copy session on tablet e00feee2662a48d8a1a57f9e52275154 from peer c2a28baa7ee2469294d94b7fe5455cff at {username='slave'} at 127.14.212.196:45367: session id = c2a28baa7ee2469294d94b7fe5455cff-e00feee2662a48d8a1a57f9e52275154
I20240502 01:54:59.609177 21878 tablet_copy_source_session.cc:205] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:59.610724 32216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00feee2662a48d8a1a57f9e52275154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:59.618924 22271 tablet_copy_source_session.cc:205] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:59.626040 32383 tablet_bootstrap.cc:492] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:59.626564 32383 tablet_bootstrap.cc:492] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513: Bootstrap complete.
I20240502 01:54:59.626871 32383 ts_tablet_manager.cc:1367] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.007s
I20240502 01:54:59.627877 32383 raft_consensus.cc:348] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:59.628201 32383 raft_consensus.cc:729] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:54:59.628496 32383 consensus_queue.cc:260] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:54:59.629084 32383 ts_tablet_manager.cc:1398] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20240502 01:54:59.629417 32382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba39439749a9486799373e173110f5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:59.629801 24888 tablet_copy_service.cc:342] P d7449bb1df214bf4a5f93e4631312f08: Request end of tablet copy session 28157bc3e678428fb5f93c077d528513-77a90f08d29a49d3b451f85cbdc828a6 received from {username='slave'} at 127.14.212.198:33761
I20240502 01:54:59.629927 24888 tablet_copy_service.cc:433] P d7449bb1df214bf4a5f93e4631312f08: ending tablet copy session 28157bc3e678428fb5f93c077d528513-77a90f08d29a49d3b451f85cbdc828a6 on tablet 77a90f08d29a49d3b451f85cbdc828a6 with peer 28157bc3e678428fb5f93c077d528513
I20240502 01:54:59.645539 25001 raft_consensus.cc:1203] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:59.672304 32216 tablet_copy_client.cc:783] T e00feee2662a48d8a1a57f9e52275154 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:59.672746 32216 tablet_copy_client.cc:647] T e00feee2662a48d8a1a57f9e52275154 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:59.681449 32382 tablet_copy_client.cc:783] T ba39439749a9486799373e173110f5c5 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:59.682286 32382 tablet_copy_client.cc:647] T ba39439749a9486799373e173110f5c5 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:59.687901 32382 tablet_copy_client.cc:523] T ba39439749a9486799373e173110f5c5 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:59.690289 24583 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:59.693120 32216 tablet_copy_client.cc:523] T e00feee2662a48d8a1a57f9e52275154 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:59.697543 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:59.699651 32382 tablet_bootstrap.cc:492] T ba39439749a9486799373e173110f5c5 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:54:59.698185 32216 tablet_bootstrap.cc:492] T e00feee2662a48d8a1a57f9e52275154 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:54:59.718436 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:54:59.731566 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:59.741897 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:59.767325 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:54:59.784415 32216 tablet_bootstrap.cc:492] T e00feee2662a48d8a1a57f9e52275154 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:59.785352 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:59.791693 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:59.792037 32216 tablet_bootstrap.cc:492] T e00feee2662a48d8a1a57f9e52275154 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap complete.
I20240502 01:54:59.792327 32216 ts_tablet_manager.cc:1367] T e00feee2662a48d8a1a57f9e52275154 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.094s user 0.006s sys 0.011s
I20240502 01:54:59.838833 32216 raft_consensus.cc:348] T e00feee2662a48d8a1a57f9e52275154 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:59.841084 32216 raft_consensus.cc:729] T e00feee2662a48d8a1a57f9e52275154 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:59.842777 32216 consensus_queue.cc:260] T e00feee2662a48d8a1a57f9e52275154 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:54:59.851666 32216 ts_tablet_manager.cc:1398] T e00feee2662a48d8a1a57f9e52275154 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.059s user 0.003s sys 0.000s
I20240502 01:54:59.857363 21878 tablet_copy_service.cc:342] P 759257ea7bef415ca698e60d82199d77: Request end of tablet copy session c2a28baa7ee2469294d94b7fe5455cff-e00feee2662a48d8a1a57f9e52275154 received from {username='slave'} at 127.14.212.196:45367
I20240502 01:54:59.857486 21878 tablet_copy_service.cc:433] P 759257ea7bef415ca698e60d82199d77: ending tablet copy session c2a28baa7ee2469294d94b7fe5455cff-e00feee2662a48d8a1a57f9e52275154 on tablet e00feee2662a48d8a1a57f9e52275154 with peer c2a28baa7ee2469294d94b7fe5455cff
I20240502 01:54:59.884387 32382 tablet_bootstrap.cc:492] T ba39439749a9486799373e173110f5c5 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:59.884850 32382 tablet_bootstrap.cc:492] T ba39439749a9486799373e173110f5c5 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap complete.
I20240502 01:54:59.885113 32382 ts_tablet_manager.cc:1367] T ba39439749a9486799373e173110f5c5 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.186s user 0.049s sys 0.000s
I20240502 01:54:59.885900 32382 raft_consensus.cc:348] T ba39439749a9486799373e173110f5c5 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:59.886139 32382 raft_consensus.cc:729] T ba39439749a9486799373e173110f5c5 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:54:59.886363 32382 consensus_queue.cc:260] T ba39439749a9486799373e173110f5c5 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:54:59.891400 32382 ts_tablet_manager.cc:1398] T ba39439749a9486799373e173110f5c5 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:54:59.892081 22271 tablet_copy_service.cc:342] P c2a28baa7ee2469294d94b7fe5455cff: Request end of tablet copy session d7449bb1df214bf4a5f93e4631312f08-ba39439749a9486799373e173110f5c5 received from {username='slave'} at 127.14.212.199:55493
I20240502 01:54:59.892215 22271 tablet_copy_service.cc:433] P c2a28baa7ee2469294d94b7fe5455cff: ending tablet copy session d7449bb1df214bf4a5f93e4631312f08-ba39439749a9486799373e173110f5c5 on tablet ba39439749a9486799373e173110f5c5 with peer d7449bb1df214bf4a5f93e4631312f08
I20240502 01:54:59.960736 22117 raft_consensus.cc:1203] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:00.019512 22251 raft_consensus.cc:1203] T e00feee2662a48d8a1a57f9e52275154 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:55:00.066357 24731 raft_consensus.cc:1203] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:00.097916 24866 raft_consensus.cc:1203] T ba39439749a9486799373e173110f5c5 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:55:00.333271 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:00.358736 24583 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:00.367050 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:00.401170 21791 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.197:36677, user_credentials={real_user=slave}} blocked reactor thread for 69467.4us
I20240502 01:55:00.421032 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:00.459321 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:00.463166 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:00.506227 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:00.507074 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:00.683763 24864 consensus_queue.cc:237] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:00.685742 22118 raft_consensus.cc:1261] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:00.685918 24734 raft_consensus.cc:1261] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:00.686374 31903 consensus_queue.cc:1035] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:00.686949 31903 consensus_queue.cc:1035] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:00.689031 32159 raft_consensus.cc:2932] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:00.689697 22118 raft_consensus.cc:2932] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:00.693971 24734 raft_consensus.cc:2932] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:00.698628 21713 catalog_manager.cc:5484] T 77a90f08d29a49d3b451f85cbdc828a6 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:00.705330 21837 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a90f08d29a49d3b451f85cbdc828a6 with delete_type TABLET_DATA_TOMBSTONED (TS 759257ea7bef415ca698e60d82199d77 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47656
I20240502 01:55:00.706039 32515 tablet_replica.cc:330] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:55:00.706349 32515 raft_consensus.cc:2229] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:00.706676 32515 raft_consensus.cc:2258] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:00.707456 32515 ts_tablet_manager.cc:1875] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:00.713796 32515 ts_tablet_manager.cc:1888] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:00.714031 32515 log.cc:1196] T 77a90f08d29a49d3b451f85cbdc828a6 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/77a90f08d29a49d3b451f85cbdc828a6
I20240502 01:55:00.714766 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 77a90f08d29a49d3b451f85cbdc828a6 (table rebalancer_extra_table_52 [id=b77487e3a60d4b7abb035e27c08af026]) successfully deleted
I20240502 01:55:01.063921 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:01.065217 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:01.084497 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:01.089329 24583 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:01.093745 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:01.097776 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:01.102052 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:01.172322 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:01.286546 21857 consensus_queue.cc:237] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:01.288928 24599 raft_consensus.cc:1261] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:01.289625 32593 consensus_queue.cc:1035] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:01.292346 32478 raft_consensus.cc:2932] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:01.293206 25000 raft_consensus.cc:1261] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:01.293234 24599 raft_consensus.cc:2932] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:01.303826 21715 catalog_manager.cc:5484] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:01.308666 32593 consensus_queue.cc:1035] T ea103b58ddcc4459af3d51878a110d38 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:01.330971 25000 raft_consensus.cc:2932] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:01.368527 24847 tablet_service.cc:1514] Processing DeleteTablet for tablet ea103b58ddcc4459af3d51878a110d38 with delete_type TABLET_DATA_TOMBSTONED (TS d7449bb1df214bf4a5f93e4631312f08 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51592
I20240502 01:55:01.377377 32620 tablet_replica.cc:330] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08: stopping tablet replica
I20240502 01:55:01.377650 32620 raft_consensus.cc:2229] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:01.378182 32620 raft_consensus.cc:2258] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:01.381203 32620 ts_tablet_manager.cc:1875] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:01.392081 32620 ts_tablet_manager.cc:1888] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:01.392369 32620 log.cc:1196] T ea103b58ddcc4459af3d51878a110d38 P d7449bb1df214bf4a5f93e4631312f08: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/wals/ea103b58ddcc4459af3d51878a110d38
I20240502 01:55:01.393496 21700 catalog_manager.cc:4830] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): tablet ea103b58ddcc4459af3d51878a110d38 (table rebalancer_extra_table_52 [id=b77487e3a60d4b7abb035e27c08af026]) successfully deleted
I20240502 01:55:01.705646 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:01.711273 24583 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:01.754029 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:01.758592 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:01.847682 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:01.866575 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:01.905457 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:01.907343 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:02.056532 24864 consensus_queue.cc:237] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:02.058764 22248 raft_consensus.cc:1261] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:02.059006 24601 raft_consensus.cc:1261] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:02.059216 32159 consensus_queue.cc:1035] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:02.059856 32694 consensus_queue.cc:1035] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:02.061789 32694 raft_consensus.cc:2932] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:02.062597 22248 raft_consensus.cc:2932] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:02.067860 24601 raft_consensus.cc:2932] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:02.068774 21715 catalog_manager.cc:5484] T 3237002dce97460799e219cf5b4968cb P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:02.082406 24715 tablet_service.cc:1514] Processing DeleteTablet for tablet 3237002dce97460799e219cf5b4968cb with delete_type TABLET_DATA_TOMBSTONED (TS 28157bc3e678428fb5f93c077d528513 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41650
I20240502 01:55:02.083108 32705 tablet_replica.cc:330] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:55:02.083820 32705 raft_consensus.cc:2229] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:02.084126 32705 raft_consensus.cc:2258] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:02.085417 32705 ts_tablet_manager.cc:1875] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:02.093735 32705 ts_tablet_manager.cc:1888] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:02.094046 32705 log.cc:1196] T 3237002dce97460799e219cf5b4968cb P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/3237002dce97460799e219cf5b4968cb
I20240502 01:55:02.095808 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet 3237002dce97460799e219cf5b4968cb (table rebalancer_extra_table_22 [id=53be2be99d0d46df8778de508abff3a3]) successfully deleted
I20240502 01:55:02.310488 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:02.311054 24583 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:02.447489 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:02.484962 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:02.510020 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240502 01:55:02.523872 24668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.193:33785, user_credentials={real_user=slave}} blocked reactor thread for 113853us
W20240502 01:55:02.533439 24670 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.195:38879, user_credentials={real_user=slave}} blocked reactor thread for 92988.8us
I20240502 01:55:02.549635 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:02.622115 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:02.715610 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:02.815536 24731 consensus_queue.cc:237] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:02.818547 24864 raft_consensus.cc:1261] T 0a1b190978514cb49efb2b97cc97a65f P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:02.818780 21856 raft_consensus.cc:1261] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:02.819458 32755 consensus_queue.cc:1035] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:02.819933 32755 consensus_queue.cc:1035] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:02.828639 32755 raft_consensus.cc:2932] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:02.832764 21856 raft_consensus.cc:2932] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:02.836216 21714 catalog_manager.cc:5484] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 reported cstate change: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:02.841867 24864 raft_consensus.cc:2932] T 0a1b190978514cb49efb2b97cc97a65f P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:02.876533 24583 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a1b190978514cb49efb2b97cc97a65f with delete_type TABLET_DATA_TOMBSTONED (TS 0c64d40b5d7644bd8b29feaa80eab72f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53832
I20240502 01:55:02.879827 325 tablet_replica.cc:330] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:55:02.880084 325 raft_consensus.cc:2229] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:02.880307 325 raft_consensus.cc:2258] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:02.886773 325 ts_tablet_manager.cc:1875] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:02.902258 325 ts_tablet_manager.cc:1888] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:02.902419 325 log.cc:1196] T 0a1b190978514cb49efb2b97cc97a65f P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/0a1b190978514cb49efb2b97cc97a65f
I20240502 01:55:02.903272 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet 0a1b190978514cb49efb2b97cc97a65f (table rebalancer_extra_table_34 [id=eda35ad172274afa81b14d7170413fac]) successfully deleted
I20240502 01:55:03.224363 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:03.237274 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:03.262524 24583 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:03.269678 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:03.342280 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:03.437739 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:03.450616 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:03.512780 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:03.626866 21986 consensus_queue.cc:237] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:03.629354 24731 raft_consensus.cc:1261] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:03.629376 24864 raft_consensus.cc:1261] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:03.629904 32758 consensus_queue.cc:1035] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:03.630476 32617 consensus_queue.cc:1035] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:03.633507 32758 raft_consensus.cc:2932] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:03.634307 24864 raft_consensus.cc:2932] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:03.638474 24734 raft_consensus.cc:2932] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:03.640967 21715 catalog_manager.cc:5484] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index 2 to 3, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:03.654793 24981 tablet_service.cc:1514] Processing DeleteTablet for tablet fafa4ef17fac41b68e3bf4aaa8a4caa8 with delete_type TABLET_DATA_TOMBSTONED (TS 41c70f373c9d49a49891b03a12c6cb2b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:39964
I20240502 01:55:03.655462 417 tablet_replica.cc:330] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:55:03.655714 417 raft_consensus.cc:2229] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:03.655967 417 raft_consensus.cc:2258] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:03.657358 417 ts_tablet_manager.cc:1875] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:03.663884 417 ts_tablet_manager.cc:1888] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:03.664073 417 log.cc:1196] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/fafa4ef17fac41b68e3bf4aaa8a4caa8
I20240502 01:55:03.665283 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet fafa4ef17fac41b68e3bf4aaa8a4caa8 (table rebalancer_extra_table_24 [id=bb7acab87c2b4608b70144063a8a4a6e]) successfully deleted
I20240502 01:55:03.952145 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:03.990984 24583 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:04.003561 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:04.016660 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:04.032354 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:04.064518 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:04.079512 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:04.109583 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:04.226351 24864 consensus_queue.cc:237] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:04.228338 22120 raft_consensus.cc:1261] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:04.228338 22248 raft_consensus.cc:1261] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:04.228914 32767 consensus_queue.cc:1035] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:04.229226 32159 consensus_queue.cc:1035] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:04.231941 32767 raft_consensus.cc:2932] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:04.232630 22248 raft_consensus.cc:2932] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:04.233400 22120 raft_consensus.cc:2932] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:04.239594 21713 catalog_manager.cc:5484] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:04.251812 24583 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ce01a3c9d9948ba8f153ad150cf3cb0 with delete_type TABLET_DATA_TOMBSTONED (TS 0c64d40b5d7644bd8b29feaa80eab72f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53832
I20240502 01:55:04.254520 498 tablet_replica.cc:330] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:55:04.254886 498 raft_consensus.cc:2229] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:04.255198 498 raft_consensus.cc:2258] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:04.256029 498 ts_tablet_manager.cc:1875] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:04.262843 498 ts_tablet_manager.cc:1888] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:04.263072 498 log.cc:1196] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/7ce01a3c9d9948ba8f153ad150cf3cb0
I20240502 01:55:04.263983 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet 7ce01a3c9d9948ba8f153ad150cf3cb0 (table rebalancer_extra_table_20 [id=adc0aa06508a406b91ce8b5a5a44ea29]) successfully deleted
I20240502 01:55:04.631464 24583 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:04.650578 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:04.663388 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:04.674860 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:04.714676 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:04.759143 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:04.791790 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:04.809181 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:05.004729 25000 consensus_queue.cc:237] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:05.006793 21857 raft_consensus.cc:1261] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:05.006835 22251 raft_consensus.cc:1261] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:05.007352 32643 consensus_queue.cc:1035] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:05.007712 32766 consensus_queue.cc:1035] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:05.010991 32643 raft_consensus.cc:2932] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:05.013191 21857 raft_consensus.cc:2932] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:05.022251 21713 catalog_manager.cc:5484] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:05.046778 22248 raft_consensus.cc:2932] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:05.060714 24715 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e8d8fbd2d1249d5b5de5f189b0d03e9 with delete_type TABLET_DATA_TOMBSTONED (TS 28157bc3e678428fb5f93c077d528513 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41650
I20240502 01:55:05.067899 594 tablet_replica.cc:330] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:55:05.068200 594 raft_consensus.cc:2229] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:05.068428 594 raft_consensus.cc:2258] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:05.069193 594 ts_tablet_manager.cc:1875] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:05.073825 594 ts_tablet_manager.cc:1888] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:05.073927 594 log.cc:1196] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/6e8d8fbd2d1249d5b5de5f189b0d03e9
I20240502 01:55:05.074604 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet 6e8d8fbd2d1249d5b5de5f189b0d03e9 (table rebalancer_extra_table_24 [id=bb7acab87c2b4608b70144063a8a4a6e]) successfully deleted
I20240502 01:55:05.276674 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:05.303563 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:05.312965 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:05.361331 24583 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:05.380769 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 01:55:05.396147 24802 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.197:36677, user_credentials={real_user=slave}} blocked reactor thread for 104189us
I20240502 01:55:05.463579 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:05.487200 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:05.500272 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:05.607851 21987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a9fb0b24337a4175972db268aac5df34"
dest_uuid: "06543b05a16f42099bc86db2a0284412"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42580
I20240502 01:55:05.608036 21987 raft_consensus.cc:595] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:05.698830 32758 raft_consensus.cc:979] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412: : Instructing follower 0c64d40b5d7644bd8b29feaa80eab72f to start an election
I20240502 01:55:05.699012 32617 raft_consensus.cc:1067] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Signalling peer 0c64d40b5d7644bd8b29feaa80eab72f to start an election
I20240502 01:55:05.700029 24599 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9fb0b24337a4175972db268aac5df34"
dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
from {username='slave'} at 127.14.212.194:42615
I20240502 01:55:05.700255 24599 raft_consensus.cc:482] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:55:05.700368 24599 raft_consensus.cc:3037] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:05.702821 24599 raft_consensus.cc:504] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:05.704639 24599 leader_election.cc:288] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:55:05.705276 24733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fb0b24337a4175972db268aac5df34" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:55:05.705518 24733 raft_consensus.cc:3037] T a9fb0b24337a4175972db268aac5df34 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:05.705896 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fb0b24337a4175972db268aac5df34" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:55:05.706151 22120 raft_consensus.cc:3037] T a9fb0b24337a4175972db268aac5df34 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:05.706421 21987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fb0b24337a4175972db268aac5df34" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:55:05.706609 21987 raft_consensus.cc:3032] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:55:05.706697 21987 raft_consensus.cc:729] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:55:05.706948 21987 consensus_queue.cc:260] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:05.707387 21987 raft_consensus.cc:3037] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:05.707808 24733 raft_consensus.cc:2454] T a9fb0b24337a4175972db268aac5df34 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 2.
I20240502 01:55:05.709009 22120 raft_consensus.cc:2454] T a9fb0b24337a4175972db268aac5df34 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 2.
I20240502 01:55:05.709619 24537 leader_election.cc:302] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:55:05.709905 645 raft_consensus.cc:2781] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:05.710047 645 raft_consensus.cc:686] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:55:05.710528 21987 raft_consensus.cc:2454] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 2.
I20240502 01:55:05.710366 645 consensus_queue.cc:237] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:05.717515 21708 catalog_manager.cc:5484] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 1 to 2, leader changed from 06543b05a16f42099bc86db2a0284412 (127.14.212.194) to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 2 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:05.958374 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:06.009447 24583 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:06.076339 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:06.130805 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:06.132728 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:06.151572 22118 raft_consensus.cc:1261] T a9fb0b24337a4175972db268aac5df34 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:55:06.152200 645 consensus_queue.cc:1035] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:06.185253 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:06.194768 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:06.210754 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:06.290194 24735 raft_consensus.cc:1261] T a9fb0b24337a4175972db268aac5df34 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:55:06.291090 716 consensus_queue.cc:1035] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:06.313905 21987 raft_consensus.cc:1261] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:55:06.314891 32484 consensus_queue.cc:1035] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:06.743186 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:06.757907 24583 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:06.773249 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:06.793056 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:06.814338 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:06.870265 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:06.933501 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:07.051183 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:07.152163 21986 consensus_queue.cc:237] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:07.154382 24603 raft_consensus.cc:1261] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:07.154548 24867 raft_consensus.cc:1261] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:07.155130 768 consensus_queue.cc:1035] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:07.155732 32758 consensus_queue.cc:1035] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:07.159054 768 raft_consensus.cc:2932] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:07.159780 24867 raft_consensus.cc:2932] T 0f8200b044ef44e1a0451baf2b1485d2 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:07.169507 21713 catalog_manager.cc:5484] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index 6 to 7, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:07.178687 24599 raft_consensus.cc:2932] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:07.202095 21837 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f8200b044ef44e1a0451baf2b1485d2 with delete_type TABLET_DATA_TOMBSTONED (TS 759257ea7bef415ca698e60d82199d77 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47656
I20240502 01:55:07.202829 828 tablet_replica.cc:330] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:55:07.203121 828 raft_consensus.cc:2229] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:07.203513 828 raft_consensus.cc:2258] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:07.204427 828 ts_tablet_manager.cc:1875] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:07.213272 828 ts_tablet_manager.cc:1888] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:07.213397 828 log.cc:1196] T 0f8200b044ef44e1a0451baf2b1485d2 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/0f8200b044ef44e1a0451baf2b1485d2
I20240502 01:55:07.214084 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 0f8200b044ef44e1a0451baf2b1485d2 (table rebalance_test_table_17 [id=7a17c6db79004eb696e7dd281fdc4f3b]) successfully deleted
I20240502 01:55:07.515916 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:07.568521 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:07.576479 24583 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:07.623418 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:07.637017 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:07.673202 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:07.706861 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:07.775553 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:07.968173 24864 consensus_queue.cc:237] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:07.970861 21986 raft_consensus.cc:1261] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:07.971259 24599 raft_consensus.cc:1261] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:07.971421 638 consensus_queue.cc:1035] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:07.971865 894 consensus_queue.cc:1035] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:07.975334 894 raft_consensus.cc:2932] T c351a8e007d04327bb14303d7b32d5be P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:07.975961 21986 raft_consensus.cc:2932] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:07.979888 24599 raft_consensus.cc:2932] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:07.986259 21713 catalog_manager.cc:5484] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index 2 to 3, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:07.997215 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet c351a8e007d04327bb14303d7b32d5be with delete_type TABLET_DATA_TOMBSTONED (TS c64840657cdf4b93980bd34120c94d9a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40366
I20240502 01:55:08.001739 926 tablet_replica.cc:330] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:55:08.002044 926 raft_consensus.cc:2229] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:08.002436 926 raft_consensus.cc:2258] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:08.003368 926 ts_tablet_manager.cc:1875] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:08.010993 926 ts_tablet_manager.cc:1888] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:08.011114 926 log.cc:1196] T c351a8e007d04327bb14303d7b32d5be P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/c351a8e007d04327bb14303d7b32d5be
I20240502 01:55:08.011884 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet c351a8e007d04327bb14303d7b32d5be (table rebalancer_extra_table_32 [id=e6f275d6ff034b158e43e06075db0828]) successfully deleted
I20240502 01:55:08.266768 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:08.313022 24583 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:08.338790 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:08.359560 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:08.359736 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:08.403841 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:08.421984 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:08.429320 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:08.628352 22248 consensus_queue.cc:237] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:08.630836 24864 raft_consensus.cc:1261] T d81f9cab671a4258aee1dd4ea4a5ee6b P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:08.631350 21856 raft_consensus.cc:1261] T d81f9cab671a4258aee1dd4ea4a5ee6b P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:08.631932 897 consensus_queue.cc:1035] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:08.633378 773 consensus_queue.cc:1035] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:08.636552 897 raft_consensus.cc:2932] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:08.637094 24864 raft_consensus.cc:2932] T d81f9cab671a4258aee1dd4ea4a5ee6b P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:08.643980 21711 catalog_manager.cc:5484] T d81f9cab671a4258aee1dd4ea4a5ee6b P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:08.655346 21856 raft_consensus.cc:2932] T d81f9cab671a4258aee1dd4ea4a5ee6b P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:08.664499 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet d81f9cab671a4258aee1dd4ea4a5ee6b with delete_type TABLET_DATA_TOMBSTONED (TS c64840657cdf4b93980bd34120c94d9a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40366
I20240502 01:55:08.682885 1018 tablet_replica.cc:330] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:55:08.683138 1018 raft_consensus.cc:2229] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:08.683437 1018 raft_consensus.cc:2258] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:08.684290 1018 ts_tablet_manager.cc:1875] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:08.690459 1018 ts_tablet_manager.cc:1888] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:55:08.690711 1018 log.cc:1196] T d81f9cab671a4258aee1dd4ea4a5ee6b P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/d81f9cab671a4258aee1dd4ea4a5ee6b
I20240502 01:55:08.692272 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet d81f9cab671a4258aee1dd4ea4a5ee6b (table rebalance_test_table_13 [id=b462b431b51b4d45baf0ff5a495aa90a]) successfully deleted
W20240502 01:55:08.912351 24536 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.193:33785, user_credentials={real_user=slave}} blocked reactor thread for 45210.4us
I20240502 01:55:09.037302 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:09.109498 24583 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:09.111917 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:09.130741 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:09.143769 22229 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:09.166743 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:09.173200 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:09.221174 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 01:55:09.331421 24803 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.195:38879, user_credentials={real_user=slave}} blocked reactor thread for 44884us
I20240502 01:55:09.415032 22251 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5d40847a56704b33b771c3fd87d6dfae"
dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51478
I20240502 01:55:09.415333 22251 raft_consensus.cc:595] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:09.700949 897 raft_consensus.cc:979] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff: : Instructing follower 41c70f373c9d49a49891b03a12c6cb2b to start an election
I20240502 01:55:09.701146 897 raft_consensus.cc:1067] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Signalling peer 41c70f373c9d49a49891b03a12c6cb2b to start an election
I20240502 01:55:09.702042 25001 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5d40847a56704b33b771c3fd87d6dfae"
dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
from {username='slave'} at 127.14.212.196:54787
I20240502 01:55:09.702265 25001 raft_consensus.cc:482] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:55:09.702366 25001 raft_consensus.cc:3037] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:09.709903 25001 raft_consensus.cc:504] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:09.710870 25001 leader_election.cc:288] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 election: Requested vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:55:09.711416 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40847a56704b33b771c3fd87d6dfae" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:55:09.711640 22251 raft_consensus.cc:3032] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:55:09.711709 22251 raft_consensus.cc:729] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:55:09.712018 22251 consensus_queue.cc:260] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:09.712468 22251 raft_consensus.cc:3037] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:09.713102 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40847a56704b33b771c3fd87d6dfae" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:55:09.713318 21854 raft_consensus.cc:3037] T 5d40847a56704b33b771c3fd87d6dfae P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:09.713315 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40847a56704b33b771c3fd87d6dfae" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:55:09.713529 22120 raft_consensus.cc:3037] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:09.716188 21854 raft_consensus.cc:2454] T 5d40847a56704b33b771c3fd87d6dfae P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 2.
I20240502 01:55:09.718724 22120 raft_consensus.cc:2454] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 2.
I20240502 01:55:09.719234 24936 leader_election.cc:302] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:55:09.719547 735 raft_consensus.cc:2781] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:09.719689 735 raft_consensus.cc:686] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b [term 2 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:55:09.719926 735 consensus_queue.cc:237] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:09.716424 22251 raft_consensus.cc:2454] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 2.
I20240502 01:55:09.757546 21709 catalog_manager.cc:5484] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 1 to 2, leader changed from c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 2 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:09.766244 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:09.780050 24981 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:09.790138 24583 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:09.798568 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:09.817292 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:09.896979 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:09.899655 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:09.946012 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 01:55:09.988090 24537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.198:42375, user_credentials={real_user=slave}} blocked reactor thread for 51944.7us
I20240502 01:55:10.105839 22120 raft_consensus.cc:1261] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:55:10.106491 735 consensus_queue.cc:1035] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:10.140532 21856 raft_consensus.cc:1261] T 5d40847a56704b33b771c3fd87d6dfae P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:55:10.141232 735 consensus_queue.cc:1035] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:10.145946 22248 raft_consensus.cc:1261] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:55:10.147564 735 consensus_queue.cc:1035] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:10.419747 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:10.449506 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:10.477878 24583 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:10.491798 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:10.526114 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:10.545647 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:10.562309 24981 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:10.694661 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:10.802369 21986 consensus_queue.cc:237] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:10.804476 25001 raft_consensus.cc:1261] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:10.805203 21856 raft_consensus.cc:1261] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:10.806054 32758 consensus_queue.cc:1035] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:10.806378 1222 consensus_queue.cc:1035] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:10.814715 1222 raft_consensus.cc:2932] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:10.815510 21854 raft_consensus.cc:2932] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:10.824384 21711 catalog_manager.cc:5484] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:10.828943 25000 raft_consensus.cc:2932] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:10.843781 24847 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d6f1c51e76342429deb0ccbf4ad8ce2 with delete_type TABLET_DATA_TOMBSTONED (TS d7449bb1df214bf4a5f93e4631312f08 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51592
I20240502 01:55:10.844522 1257 tablet_replica.cc:330] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08: stopping tablet replica
I20240502 01:55:10.844821 1257 raft_consensus.cc:2229] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:10.845155 1257 raft_consensus.cc:2258] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:10.845993 1257 ts_tablet_manager.cc:1875] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:10.857805 1257 ts_tablet_manager.cc:1888] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:10.858060 1257 log.cc:1196] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P d7449bb1df214bf4a5f93e4631312f08: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/wals/2d6f1c51e76342429deb0ccbf4ad8ce2
I20240502 01:55:10.859167 21700 catalog_manager.cc:4830] TS d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): tablet 2d6f1c51e76342429deb0ccbf4ad8ce2 (table rebalancer_extra_table_42 [id=64a3018c59994b83b4283de8ca4e1f27]) successfully deleted
I20240502 01:55:11.142511 24583 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:11.152459 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:11.183197 24981 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:11.191506 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:11.215229 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:11.249912 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:11.254932 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:11.284188 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:11.441012 21854 consensus_queue.cc:237] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:11.443203 22249 raft_consensus.cc:1261] T e00feee2662a48d8a1a57f9e52275154 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:11.443212 22120 raft_consensus.cc:1261] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:11.443887 32478 consensus_queue.cc:1035] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:11.444211 1227 consensus_queue.cc:1035] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:11.447467 1227 raft_consensus.cc:2932] T e00feee2662a48d8a1a57f9e52275154 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:11.448288 22251 raft_consensus.cc:2932] T e00feee2662a48d8a1a57f9e52275154 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:11.451495 22120 raft_consensus.cc:2932] T e00feee2662a48d8a1a57f9e52275154 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:11.454768 21714 catalog_manager.cc:5484] T e00feee2662a48d8a1a57f9e52275154 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index 4 to 5, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:11.480347 24981 tablet_service.cc:1514] Processing DeleteTablet for tablet e00feee2662a48d8a1a57f9e52275154 with delete_type TABLET_DATA_TOMBSTONED (TS 41c70f373c9d49a49891b03a12c6cb2b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:39964
I20240502 01:55:11.483007 1338 tablet_replica.cc:330] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:55:11.483341 1338 raft_consensus.cc:2229] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:11.483695 1338 raft_consensus.cc:2258] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:11.484405 1338 ts_tablet_manager.cc:1875] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:11.493716 1338 ts_tablet_manager.cc:1888] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:55:11.493957 1338 log.cc:1196] T e00feee2662a48d8a1a57f9e52275154 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/e00feee2662a48d8a1a57f9e52275154
I20240502 01:55:11.510934 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet e00feee2662a48d8a1a57f9e52275154 (table rebalancer_extra_table_8 [id=de52a7578d964d1ba0e315c9f1e99b21]) successfully deleted
I20240502 01:55:11.908476 24583 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:11.933585 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:11.936862 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:11.966897 24981 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:12.000380 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:12.040824 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:12.072510 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:12.132440 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:12.236181 24733 consensus_queue.cc:237] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:12.238338 24864 raft_consensus.cc:1261] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:12.238376 22118 raft_consensus.cc:1261] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:12.238929 898 consensus_queue.cc:1035] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:12.239269 899 consensus_queue.cc:1035] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:12.242048 898 raft_consensus.cc:2932] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:12.242647 22120 raft_consensus.cc:2932] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:12.247433 24864 raft_consensus.cc:2932] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:12.249190 21710 catalog_manager.cc:5484] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index 2 to 3, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:12.258109 22229 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ad2f9c27d284098a8e4f20ebc4b3d8c with delete_type TABLET_DATA_TOMBSTONED (TS c2a28baa7ee2469294d94b7fe5455cff not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50898
I20240502 01:55:12.258935 1437 tablet_replica.cc:330] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:55:12.259215 1437 raft_consensus.cc:2229] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:12.259487 1437 raft_consensus.cc:2258] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:12.260329 1437 ts_tablet_manager.cc:1875] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:12.266497 1437 ts_tablet_manager.cc:1888] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:12.266629 1437 log.cc:1196] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/9ad2f9c27d284098a8e4f20ebc4b3d8c
I20240502 01:55:12.267437 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 9ad2f9c27d284098a8e4f20ebc4b3d8c (table rebalancer_extra_table_46 [id=048399628e3c4ff99f3a5d3a67b8ff83]) successfully deleted
I20240502 01:55:12.354990 24733 consensus_queue.cc:237] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:12.358346 22120 raft_consensus.cc:1261] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:12.358556 24601 raft_consensus.cc:1261] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:12.359045 898 consensus_queue.cc:1035] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:12.359539 898 consensus_queue.cc:1035] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:55:12.360056 24668 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00c38c0987ed469eba1fb6f76ee2f6a9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:12.374028 898 raft_consensus.cc:2932] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:12.374709 24601 raft_consensus.cc:2932] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:12.381554 21710 catalog_manager.cc:5484] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:12.382380 22120 raft_consensus.cc:2932] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:12.419229 22251 consensus_queue.cc:237] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:12.422415 21854 raft_consensus.cc:1261] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:12.422874 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a2f4ea104d4409383e5eb32ca2f78b0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:12.423111 1226 consensus_queue.cc:1035] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:12.423669 22118 raft_consensus.cc:1261] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:12.424394 897 consensus_queue.cc:1035] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:12.438215 897 raft_consensus.cc:2932] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:12.439054 22118 raft_consensus.cc:2932] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:12.442289 21854 raft_consensus.cc:2932] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:12.444649 21709 catalog_manager.cc:5484] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:12.488533 22118 consensus_queue.cc:237] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:12.491294 25000 raft_consensus.cc:1261] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:12.491833 21854 raft_consensus.cc:1261] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:12.491854 1411 consensus_queue.cc:1035] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:55:12.492195 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: daaadb773202454894890c3379c4a58d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:12.492561 1411 consensus_queue.cc:1035] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:12.505789 1202 raft_consensus.cc:2932] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:12.506464 21854 raft_consensus.cc:2932] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:12.512658 24999 raft_consensus.cc:2932] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:12.518267 21710 catalog_manager.cc:5484] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index 5 to 6, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:12.788270 1545 ts_tablet_manager.cc:902] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412: Initiating tablet copy from peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:12.895695 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:12.898792 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:12.901315 24981 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:12.928661 1545 tablet_copy_client.cc:308] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.198:42375
I20240502 01:55:12.935741 24583 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:12.956135 1568 ts_tablet_manager.cc:902] T daaadb773202454894890c3379c4a58d P 28157bc3e678428fb5f93c077d528513: Initiating tablet copy from peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:55:12.957983 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:12.961884 24755 tablet_copy_service.cc:140] P 28157bc3e678428fb5f93c077d528513: Received BeginTabletCopySession request for tablet 00c38c0987ed469eba1fb6f76ee2f6a9 from peer 06543b05a16f42099bc86db2a0284412 ({username='slave'} at 127.14.212.194:45255)
I20240502 01:55:12.962070 24755 tablet_copy_service.cc:161] P 28157bc3e678428fb5f93c077d528513: Beginning new tablet copy session on tablet 00c38c0987ed469eba1fb6f76ee2f6a9 from peer 06543b05a16f42099bc86db2a0284412 at {username='slave'} at 127.14.212.194:45255: session id = 06543b05a16f42099bc86db2a0284412-00c38c0987ed469eba1fb6f76ee2f6a9
I20240502 01:55:12.965212 24755 tablet_copy_source_session.cc:205] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:12.967630 1545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c38c0987ed469eba1fb6f76ee2f6a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:12.979321 1567 ts_tablet_manager.cc:902] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 28157bc3e678428fb5f93c077d528513: Initiating tablet copy from peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:55:13.006691 1545 tablet_copy_client.cc:783] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:13.006979 1545 tablet_copy_client.cc:647] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:13.009254 1567 tablet_copy_client.cc:308] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 28157bc3e678428fb5f93c077d528513: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.196:43093
I20240502 01:55:13.009994 22271 tablet_copy_service.cc:140] P c2a28baa7ee2469294d94b7fe5455cff: Received BeginTabletCopySession request for tablet 3a2f4ea104d4409383e5eb32ca2f78b0 from peer 28157bc3e678428fb5f93c077d528513 ({username='slave'} at 127.14.212.198:56765)
I20240502 01:55:13.010155 22271 tablet_copy_service.cc:161] P c2a28baa7ee2469294d94b7fe5455cff: Beginning new tablet copy session on tablet 3a2f4ea104d4409383e5eb32ca2f78b0 from peer 28157bc3e678428fb5f93c077d528513 at {username='slave'} at 127.14.212.198:56765: session id = 28157bc3e678428fb5f93c077d528513-3a2f4ea104d4409383e5eb32ca2f78b0
I20240502 01:55:13.012241 1545 tablet_copy_client.cc:523] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:13.016379 22271 tablet_copy_source_session.cc:205] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:13.021549 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:13.019014 1567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a2f4ea104d4409383e5eb32ca2f78b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:13.047245 1567 tablet_copy_client.cc:783] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:13.047550 1567 tablet_copy_client.cc:647] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:13.047816 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:13.078907 1545 tablet_bootstrap.cc:492] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:55:13.087610 1567 tablet_copy_client.cc:523] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 28157bc3e678428fb5f93c077d528513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:13.087754 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:13.117237 1568 tablet_copy_client.cc:308] T daaadb773202454894890c3379c4a58d P 28157bc3e678428fb5f93c077d528513: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.195:38879
I20240502 01:55:13.118077 22140 tablet_copy_service.cc:140] P c64840657cdf4b93980bd34120c94d9a: Received BeginTabletCopySession request for tablet daaadb773202454894890c3379c4a58d from peer 28157bc3e678428fb5f93c077d528513 ({username='slave'} at 127.14.212.198:48493)
I20240502 01:55:13.118245 22140 tablet_copy_service.cc:161] P c64840657cdf4b93980bd34120c94d9a: Beginning new tablet copy session on tablet daaadb773202454894890c3379c4a58d from peer 28157bc3e678428fb5f93c077d528513 at {username='slave'} at 127.14.212.198:48493: session id = 28157bc3e678428fb5f93c077d528513-daaadb773202454894890c3379c4a58d
I20240502 01:55:13.121270 22140 tablet_copy_source_session.cc:205] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:13.122846 1568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daaadb773202454894890c3379c4a58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:13.123314 1545 tablet_bootstrap.cc:492] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:13.123766 1545 tablet_bootstrap.cc:492] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412: Bootstrap complete.
I20240502 01:55:13.124035 1545 ts_tablet_manager.cc:1367] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.001s
I20240502 01:55:13.124828 1545 raft_consensus.cc:348] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:13.125110 1545 raft_consensus.cc:729] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:55:13.125444 1545 consensus_queue.cc:260] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:13.133738 1545 ts_tablet_manager.cc:1398] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.010s user 0.002s sys 0.002s
I20240502 01:55:13.134877 24755 tablet_copy_service.cc:342] P 28157bc3e678428fb5f93c077d528513: Request end of tablet copy session 06543b05a16f42099bc86db2a0284412-00c38c0987ed469eba1fb6f76ee2f6a9 received from {username='slave'} at 127.14.212.194:45255
I20240502 01:55:13.135002 24755 tablet_copy_service.cc:433] P 28157bc3e678428fb5f93c077d528513: ending tablet copy session 06543b05a16f42099bc86db2a0284412-00c38c0987ed469eba1fb6f76ee2f6a9 on tablet 00c38c0987ed469eba1fb6f76ee2f6a9 with peer 06543b05a16f42099bc86db2a0284412
I20240502 01:55:13.140501 1568 tablet_copy_client.cc:783] T daaadb773202454894890c3379c4a58d P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:13.140870 1568 tablet_copy_client.cc:647] T daaadb773202454894890c3379c4a58d P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:13.141363 1567 tablet_bootstrap.cc:492] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:55:13.142658 1568 tablet_copy_client.cc:523] T daaadb773202454894890c3379c4a58d P 28157bc3e678428fb5f93c077d528513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:13.145804 1568 tablet_bootstrap.cc:492] T daaadb773202454894890c3379c4a58d P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:55:13.209422 21986 raft_consensus.cc:1203] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:13.265671 1567 tablet_bootstrap.cc:492] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 28157bc3e678428fb5f93c077d528513: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:13.266253 1567 tablet_bootstrap.cc:492] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 28157bc3e678428fb5f93c077d528513: Bootstrap complete.
I20240502 01:55:13.266602 1567 ts_tablet_manager.cc:1367] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.125s user 0.043s sys 0.005s
I20240502 01:55:13.267443 1567 raft_consensus.cc:348] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:13.267786 1567 raft_consensus.cc:729] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:55:13.268106 1567 consensus_queue.cc:260] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:13.273169 1567 ts_tablet_manager.cc:1398] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:55:13.275610 22271 tablet_copy_service.cc:342] P c2a28baa7ee2469294d94b7fe5455cff: Request end of tablet copy session 28157bc3e678428fb5f93c077d528513-3a2f4ea104d4409383e5eb32ca2f78b0 received from {username='slave'} at 127.14.212.198:56765
I20240502 01:55:13.275761 22271 tablet_copy_service.cc:433] P c2a28baa7ee2469294d94b7fe5455cff: ending tablet copy session 28157bc3e678428fb5f93c077d528513-3a2f4ea104d4409383e5eb32ca2f78b0 on tablet 3a2f4ea104d4409383e5eb32ca2f78b0 with peer 28157bc3e678428fb5f93c077d528513
I20240502 01:55:13.281271 1568 tablet_bootstrap.cc:492] T daaadb773202454894890c3379c4a58d P 28157bc3e678428fb5f93c077d528513: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:13.281726 1568 tablet_bootstrap.cc:492] T daaadb773202454894890c3379c4a58d P 28157bc3e678428fb5f93c077d528513: Bootstrap complete.
I20240502 01:55:13.281988 1568 ts_tablet_manager.cc:1367] T daaadb773202454894890c3379c4a58d P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.136s user 0.040s sys 0.006s
I20240502 01:55:13.282744 1568 raft_consensus.cc:348] T daaadb773202454894890c3379c4a58d P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:13.282963 1568 raft_consensus.cc:729] T daaadb773202454894890c3379c4a58d P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:55:13.283183 1568 consensus_queue.cc:260] T daaadb773202454894890c3379c4a58d P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:13.283727 1568 ts_tablet_manager.cc:1398] T daaadb773202454894890c3379c4a58d P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:55:13.284662 22140 tablet_copy_service.cc:342] P c64840657cdf4b93980bd34120c94d9a: Request end of tablet copy session 28157bc3e678428fb5f93c077d528513-daaadb773202454894890c3379c4a58d received from {username='slave'} at 127.14.212.198:48493
I20240502 01:55:13.284762 22140 tablet_copy_service.cc:433] P c64840657cdf4b93980bd34120c94d9a: ending tablet copy session 28157bc3e678428fb5f93c077d528513-daaadb773202454894890c3379c4a58d on tablet daaadb773202454894890c3379c4a58d with peer 28157bc3e678428fb5f93c077d528513
I20240502 01:55:13.493278 24735 raft_consensus.cc:1203] T daaadb773202454894890c3379c4a58d P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 01:55:13.517617 24735 raft_consensus.cc:1203] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 01:55:13.572737 24537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.198:42375, user_credentials={real_user=slave}} blocked reactor thread for 60988us
I20240502 01:55:13.693167 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:13.695518 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:13.719959 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:13.725906 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:13.727443 24981 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:13.732390 24583 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:13.733379 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:13.764451 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:13.996115 22249 consensus_queue.cc:237] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:13.998241 24735 raft_consensus.cc:1261] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:13.998255 22120 raft_consensus.cc:1261] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:13.998821 1626 consensus_queue.cc:1035] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:13.999068 897 consensus_queue.cc:1035] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:14.002967 1626 raft_consensus.cc:2932] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:14.003809 22120 raft_consensus.cc:2932] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:14.004495 24735 raft_consensus.cc:2932] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:14.012743 21707 catalog_manager.cc:5484] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index 4 to 5, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:14.020509 21837 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a2f4ea104d4409383e5eb32ca2f78b0 with delete_type TABLET_DATA_TOMBSTONED (TS 759257ea7bef415ca698e60d82199d77 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47656
I20240502 01:55:14.022785 1672 tablet_replica.cc:330] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:55:14.023079 1672 raft_consensus.cc:2229] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:14.023375 1672 raft_consensus.cc:2258] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:14.024199 1672 ts_tablet_manager.cc:1875] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:14.030397 1672 ts_tablet_manager.cc:1888] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:55:14.030556 1672 log.cc:1196] T 3a2f4ea104d4409383e5eb32ca2f78b0 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/3a2f4ea104d4409383e5eb32ca2f78b0
I20240502 01:55:14.031421 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 3a2f4ea104d4409383e5eb32ca2f78b0 (table rebalance_test_table_19 [id=9827c9aba79e4ba094c6e9c0e32f36f4]) successfully deleted
I20240502 01:55:14.264569 24583 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:14.293694 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:14.329157 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:14.329689 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:14.357211 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:14.369170 24981 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:14.379158 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:14.460078 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:14.614135 24733 consensus_queue.cc:237] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:14.616362 22120 raft_consensus.cc:1261] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:14.616420 21986 raft_consensus.cc:1261] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:14.616972 898 consensus_queue.cc:1035] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:14.617241 1464 consensus_queue.cc:1035] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:14.620190 898 raft_consensus.cc:2932] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:14.620961 21986 raft_consensus.cc:2932] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:14.620944 22120 raft_consensus.cc:2932] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:14.628731 21710 catalog_manager.cc:5484] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:14.635003 24583 tablet_service.cc:1514] Processing DeleteTablet for tablet 00c38c0987ed469eba1fb6f76ee2f6a9 with delete_type TABLET_DATA_TOMBSTONED (TS 0c64d40b5d7644bd8b29feaa80eab72f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53832
I20240502 01:55:14.635702 1751 tablet_replica.cc:330] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:55:14.635911 1751 raft_consensus.cc:2229] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:14.636117 1751 raft_consensus.cc:2258] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:14.636848 1751 ts_tablet_manager.cc:1875] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:14.644418 1751 ts_tablet_manager.cc:1888] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:14.644660 1751 log.cc:1196] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/00c38c0987ed469eba1fb6f76ee2f6a9
I20240502 01:55:14.645714 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet 00c38c0987ed469eba1fb6f76ee2f6a9 (table rebalancer_extra_table_22 [id=53be2be99d0d46df8778de508abff3a3]) successfully deleted
I20240502 01:55:15.006436 24981 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:15.016661 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:15.036842 24583 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:15.043452 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:15.050287 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:15.053848 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:15.083366 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:15.113857 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:15.234879 22251 consensus_queue.cc:237] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:15.237077 24867 raft_consensus.cc:1261] T ba39439749a9486799373e173110f5c5 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:15.237275 22120 raft_consensus.cc:1261] T ba39439749a9486799373e173110f5c5 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:15.237620 897 consensus_queue.cc:1035] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:15.237888 1626 consensus_queue.cc:1035] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:15.241919 1626 raft_consensus.cc:2932] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:15.242588 22120 raft_consensus.cc:2932] T ba39439749a9486799373e173110f5c5 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:15.245718 24867 raft_consensus.cc:2932] T ba39439749a9486799373e173110f5c5 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:15.251276 21709 catalog_manager.cc:5484] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:15.275777 21969 tablet_service.cc:1514] Processing DeleteTablet for tablet ba39439749a9486799373e173110f5c5 with delete_type TABLET_DATA_TOMBSTONED (TS 06543b05a16f42099bc86db2a0284412 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46720
I20240502 01:55:15.276857 1848 tablet_replica.cc:330] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:55:15.277165 1848 raft_consensus.cc:2229] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:15.277474 1848 raft_consensus.cc:2258] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:15.278301 1848 ts_tablet_manager.cc:1875] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:15.285573 1848 ts_tablet_manager.cc:1888] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:55:15.286087 1848 log.cc:1196] T ba39439749a9486799373e173110f5c5 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/ba39439749a9486799373e173110f5c5
I20240502 01:55:15.286890 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet ba39439749a9486799373e173110f5c5 (table rebalance_test_table_7 [id=cd297aa2796f428db6b815990620e2a4]) successfully deleted
I20240502 01:55:15.709131 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:15.720078 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:15.725291 24583 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:15.751760 24981 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:15.779662 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:15.799197 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:15.819766 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:15.832562 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:16.032491 24599 consensus_queue.cc:237] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:16.034642 22118 raft_consensus.cc:1261] T a9fb0b24337a4175972db268aac5df34 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:55:16.034657 24735 raft_consensus.cc:1261] T a9fb0b24337a4175972db268aac5df34 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:55:16.035295 1413 consensus_queue.cc:1035] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:16.035602 1625 consensus_queue.cc:1035] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:16.038803 1413 raft_consensus.cc:2932] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:16.044015 22118 raft_consensus.cc:2932] T a9fb0b24337a4175972db268aac5df34 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:16.047384 21713 catalog_manager.cc:5484] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: config changed from index 2 to 4, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New cstate: current_term: 2 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:16.054463 24735 raft_consensus.cc:2932] T a9fb0b24337a4175972db268aac5df34 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:16.065227 21969 tablet_service.cc:1514] Processing DeleteTablet for tablet a9fb0b24337a4175972db268aac5df34 with delete_type TABLET_DATA_TOMBSTONED (TS 06543b05a16f42099bc86db2a0284412 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46720
I20240502 01:55:16.068074 1962 tablet_replica.cc:330] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:55:16.068390 1962 raft_consensus.cc:2229] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:16.068713 1962 raft_consensus.cc:2258] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:16.070657 1962 ts_tablet_manager.cc:1875] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:16.075892 1962 ts_tablet_manager.cc:1888] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 01:55:16.076081 1962 log.cc:1196] T a9fb0b24337a4175972db268aac5df34 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/a9fb0b24337a4175972db268aac5df34
I20240502 01:55:16.076992 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet a9fb0b24337a4175972db268aac5df34 (table rebalancer_extra_table_24 [id=bb7acab87c2b4608b70144063a8a4a6e]) successfully deleted
I20240502 01:55:16.359350 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:16.367717 24583 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:16.418686 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:16.444379 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:16.459131 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:16.465410 24981 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:16.481487 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:16.521342 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:16.690891 25000 consensus_queue.cc:237] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:16.698242 22120 raft_consensus.cc:1261] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:55:16.698401 21854 raft_consensus.cc:1261] T 5d40847a56704b33b771c3fd87d6dfae P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:55:16.698851 1537 consensus_queue.cc:1035] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:16.699503 1537 consensus_queue.cc:1035] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:16.702464 1537 raft_consensus.cc:2932] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:16.703650 22120 raft_consensus.cc:2932] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:16.705277 21854 raft_consensus.cc:2932] T 5d40847a56704b33b771c3fd87d6dfae P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:16.710048 21715 catalog_manager.cc:5484] T 5d40847a56704b33b771c3fd87d6dfae P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index 4 to 6, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New cstate: current_term: 2 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:16.717064 22229 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d40847a56704b33b771c3fd87d6dfae with delete_type TABLET_DATA_TOMBSTONED (TS c2a28baa7ee2469294d94b7fe5455cff not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50898
I20240502 01:55:16.720788 2055 tablet_replica.cc:330] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:55:16.721062 2055 raft_consensus.cc:2229] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:16.721261 2055 raft_consensus.cc:2258] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:16.721966 2055 ts_tablet_manager.cc:1875] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:16.730857 2055 ts_tablet_manager.cc:1888] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 01:55:16.730988 2055 log.cc:1196] T 5d40847a56704b33b771c3fd87d6dfae P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/5d40847a56704b33b771c3fd87d6dfae
I20240502 01:55:16.731824 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 5d40847a56704b33b771c3fd87d6dfae (table rebalance_test_table_7 [id=cd297aa2796f428db6b815990620e2a4]) successfully deleted
I20240502 01:55:16.785604 21856 consensus_queue.cc:237] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
W20240502 01:55:16.788609 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1c6c622638444bd854616902b8d91ad. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:16.788727 25000 raft_consensus.cc:1261] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:16.788955 24603 raft_consensus.cc:1261] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:16.789299 2087 consensus_queue.cc:1035] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:16.789532 1710 consensus_queue.cc:1035] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:16.798142 1710 raft_consensus.cc:2932] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:16.798985 24603 raft_consensus.cc:2932] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:16.805405 21714 catalog_manager.cc:5484] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:16.805903 25000 raft_consensus.cc:2932] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:16.943429 25000 consensus_queue.cc:237] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:16.954344 21985 raft_consensus.cc:1261] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:16.954445 21856 raft_consensus.cc:1261] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:16.955073 24936 consensus_peers.cc:485] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Couldn't send request to peer c2a28baa7ee2469294d94b7fe5455cff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 992544bbea6547de96298fd32d74b232. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:16.955282 1298 consensus_queue.cc:1035] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:16.955881 1298 consensus_queue.cc:1035] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:16.959744 1298 raft_consensus.cc:2932] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:16.961412 21985 raft_consensus.cc:2932] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:16.966832 21715 catalog_manager.cc:5484] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: config changed from index 3 to 4, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:16.968343 21856 raft_consensus.cc:2932] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:17.010828 21985 consensus_queue.cc:237] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:17.013906 22118 raft_consensus.cc:1261] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:17.013907 21856 raft_consensus.cc:1261] T 8a0ae0d8e8784db9988b3469d318f7fb P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:17.014600 1303 consensus_queue.cc:1035] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:55:17.015261 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a0ae0d8e8784db9988b3469d318f7fb. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:17.015427 2127 consensus_queue.cc:1035] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:17.028471 2128 raft_consensus.cc:2932] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:17.029126 21856 raft_consensus.cc:2932] T 8a0ae0d8e8784db9988b3469d318f7fb P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:17.032945 22118 raft_consensus.cc:2932] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:17.035713 21713 catalog_manager.cc:5484] T 8a0ae0d8e8784db9988b3469d318f7fb P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:17.076100 24603 consensus_queue.cc:237] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:17.086745 21856 raft_consensus.cc:1261] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:55:17.087507 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0877e287cb94abf98723449b9805f80. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:17.087742 1413 consensus_queue.cc:1035] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:17.089042 22251 raft_consensus.cc:1261] T c0877e287cb94abf98723449b9805f80 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:17.092067 1413 consensus_queue.cc:1035] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:17.096225 2148 raft_consensus.cc:2932] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:17.096976 22251 raft_consensus.cc:2932] T c0877e287cb94abf98723449b9805f80 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:17.100661 21856 raft_consensus.cc:2932] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:17.103389 21714 catalog_manager.cc:5484] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:17.139214 24732 consensus_queue.cc:237] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:17.142099 22118 raft_consensus.cc:1261] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:17.142194 24603 raft_consensus.cc:1261] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:17.142774 898 consensus_queue.cc:1035] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:55:17.143741 24671 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 -> Peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Couldn't send request to peer 41c70f373c9d49a49891b03a12c6cb2b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0cce126a8ba64fb2b40c4ddd03982f19. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:17.143817 898 consensus_queue.cc:1035] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:17.146616 2026 raft_consensus.cc:2932] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:17.147225 24603 raft_consensus.cc:2932] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:17.154829 21715 catalog_manager.cc:5484] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: config changed from index -1 to 2, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:17.154796 22118 raft_consensus.cc:2932] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:17.192184 24603 consensus_queue.cc:237] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:17.194933 24732 raft_consensus.cc:1261] T 5152c0bcae6946059900d52f0d33ac5b P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:17.195143 22251 raft_consensus.cc:1261] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:17.195613 1413 consensus_queue.cc:1035] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:55:17.195888 24536 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5152c0bcae6946059900d52f0d33ac5b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:17.196254 1413 consensus_queue.cc:1035] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:17.200268 2148 raft_consensus.cc:2932] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:17.200999 24735 raft_consensus.cc:2932] T 5152c0bcae6946059900d52f0d33ac5b P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:17.204174 22251 raft_consensus.cc:2932] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:17.207118 21713 catalog_manager.cc:5484] T 5152c0bcae6946059900d52f0d33ac5b P 28157bc3e678428fb5f93c077d528513 reported cstate change: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:17.245400 21854 consensus_queue.cc:237] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:17.248083 22251 raft_consensus.cc:1261] T 3be66f4b90824a21a5f41915c9041970 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:17.248229 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3be66f4b90824a21a5f41915c9041970. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:17.248250 22118 raft_consensus.cc:1261] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:17.248713 2090 consensus_queue.cc:1035] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:17.249037 1710 consensus_queue.cc:1035] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:17.254602 2090 raft_consensus.cc:2932] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:17.255008 22118 raft_consensus.cc:2932] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:17.261094 22251 raft_consensus.cc:2932] T 3be66f4b90824a21a5f41915c9041970 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:17.262769 21715 catalog_manager.cc:5484] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index -1 to 4, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:17.294988 2213 ts_tablet_manager.cc:902] T b1c6c622638444bd854616902b8d91ad P d7449bb1df214bf4a5f93e4631312f08: Initiating tablet copy from peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:55:17.296499 2213 tablet_copy_client.cc:308] T b1c6c622638444bd854616902b8d91ad P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.193:33785
I20240502 01:55:17.297217 21878 tablet_copy_service.cc:140] P 759257ea7bef415ca698e60d82199d77: Received BeginTabletCopySession request for tablet b1c6c622638444bd854616902b8d91ad from peer d7449bb1df214bf4a5f93e4631312f08 ({username='slave'} at 127.14.212.199:41477)
I20240502 01:55:17.297369 21878 tablet_copy_service.cc:161] P 759257ea7bef415ca698e60d82199d77: Beginning new tablet copy session on tablet b1c6c622638444bd854616902b8d91ad from peer d7449bb1df214bf4a5f93e4631312f08 at {username='slave'} at 127.14.212.199:41477: session id = d7449bb1df214bf4a5f93e4631312f08-b1c6c622638444bd854616902b8d91ad
I20240502 01:55:17.300285 21878 tablet_copy_source_session.cc:205] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:17.302083 2213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c6c622638444bd854616902b8d91ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:17.311439 2213 tablet_copy_client.cc:783] T b1c6c622638444bd854616902b8d91ad P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:17.311792 2213 tablet_copy_client.cc:647] T b1c6c622638444bd854616902b8d91ad P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:17.317883 2213 tablet_copy_client.cc:523] T b1c6c622638444bd854616902b8d91ad P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:17.321521 2213 tablet_bootstrap.cc:492] T b1c6c622638444bd854616902b8d91ad P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:55:17.337406 2213 tablet_bootstrap.cc:492] T b1c6c622638444bd854616902b8d91ad P d7449bb1df214bf4a5f93e4631312f08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:17.337828 2213 tablet_bootstrap.cc:492] T b1c6c622638444bd854616902b8d91ad P d7449bb1df214bf4a5f93e4631312f08: Bootstrap complete.
I20240502 01:55:17.338083 2213 ts_tablet_manager.cc:1367] T b1c6c622638444bd854616902b8d91ad P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.003s
I20240502 01:55:17.338887 2213 raft_consensus.cc:348] T b1c6c622638444bd854616902b8d91ad P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:17.339124 2213 raft_consensus.cc:729] T b1c6c622638444bd854616902b8d91ad P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:55:17.339404 2213 consensus_queue.cc:260] T b1c6c622638444bd854616902b8d91ad P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:17.340178 2213 ts_tablet_manager.cc:1398] T b1c6c622638444bd854616902b8d91ad P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:55:17.341112 21878 tablet_copy_service.cc:342] P 759257ea7bef415ca698e60d82199d77: Request end of tablet copy session d7449bb1df214bf4a5f93e4631312f08-b1c6c622638444bd854616902b8d91ad received from {username='slave'} at 127.14.212.199:41477
I20240502 01:55:17.341279 21878 tablet_copy_service.cc:433] P 759257ea7bef415ca698e60d82199d77: ending tablet copy session d7449bb1df214bf4a5f93e4631312f08-b1c6c622638444bd854616902b8d91ad on tablet b1c6c622638444bd854616902b8d91ad with peer d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:17.391500 22249 consensus_queue.cc:237] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:17.411454 21985 raft_consensus.cc:1261] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:17.411788 22120 raft_consensus.cc:1261] T 3a27dc6af117475cb227e3526d2a9007 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:17.411900 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a27dc6af117475cb227e3526d2a9007. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:17.412447 1626 consensus_queue.cc:1035] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:17.412833 2033 consensus_queue.cc:1035] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:17.420114 1626 raft_consensus.cc:2932] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:17.421097 22120 raft_consensus.cc:2932] T 3a27dc6af117475cb227e3526d2a9007 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:17.429394 21709 catalog_manager.cc:5484] T 3a27dc6af117475cb227e3526d2a9007 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:17.431350 21985 raft_consensus.cc:2932] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:17.469630 2247 ts_tablet_manager.cc:902] T 992544bbea6547de96298fd32d74b232 P c2a28baa7ee2469294d94b7fe5455cff: Initiating tablet copy from peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:55:17.472694 2247 tablet_copy_client.cc:308] T 992544bbea6547de96298fd32d74b232 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.200:37097
I20240502 01:55:17.473174 24999 consensus_queue.cc:237] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:17.473544 25021 tablet_copy_service.cc:140] P 41c70f373c9d49a49891b03a12c6cb2b: Received BeginTabletCopySession request for tablet 992544bbea6547de96298fd32d74b232 from peer c2a28baa7ee2469294d94b7fe5455cff ({username='slave'} at 127.14.212.196:54787)
I20240502 01:55:17.473675 25021 tablet_copy_service.cc:161] P 41c70f373c9d49a49891b03a12c6cb2b: Beginning new tablet copy session on tablet 992544bbea6547de96298fd32d74b232 from peer c2a28baa7ee2469294d94b7fe5455cff at {username='slave'} at 127.14.212.196:54787: session id = c2a28baa7ee2469294d94b7fe5455cff-992544bbea6547de96298fd32d74b232
I20240502 01:55:17.476197 25021 tablet_copy_source_session.cc:205] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:17.476470 22251 raft_consensus.cc:1261] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:17.477056 21854 raft_consensus.cc:1261] T 469abc1f70fc456b998461bd367dfc84 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:17.477732 2247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 992544bbea6547de96298fd32d74b232. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:17.477697 2109 consensus_queue.cc:1035] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:55:17.478500 24934 consensus_peers.cc:485] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Couldn't send request to peer 0c64d40b5d7644bd8b29feaa80eab72f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 469abc1f70fc456b998461bd367dfc84. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:17.479176 1537 consensus_queue.cc:1035] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:17.484051 2247 tablet_copy_client.cc:783] T 992544bbea6547de96298fd32d74b232 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:17.484359 2247 tablet_copy_client.cc:647] T 992544bbea6547de96298fd32d74b232 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:17.486869 2247 tablet_copy_client.cc:523] T 992544bbea6547de96298fd32d74b232 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:17.493644 2254 ts_tablet_manager.cc:902] T 8a0ae0d8e8784db9988b3469d318f7fb P 28157bc3e678428fb5f93c077d528513: Initiating tablet copy from peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:55:17.493856 2247 tablet_bootstrap.cc:492] T 992544bbea6547de96298fd32d74b232 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:55:17.493983 1537 raft_consensus.cc:2932] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:17.499389 22251 raft_consensus.cc:2932] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:17.506891 21714 catalog_manager.cc:5484] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index -1 to 2, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:17.510164 21854 raft_consensus.cc:2932] T 469abc1f70fc456b998461bd367dfc84 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:17.518033 2247 tablet_bootstrap.cc:492] T 992544bbea6547de96298fd32d74b232 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:17.518494 2247 tablet_bootstrap.cc:492] T 992544bbea6547de96298fd32d74b232 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap complete.
I20240502 01:55:17.518784 2247 ts_tablet_manager.cc:1367] T 992544bbea6547de96298fd32d74b232 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.004s
I20240502 01:55:17.519464 2247 raft_consensus.cc:348] T 992544bbea6547de96298fd32d74b232 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:17.519816 2247 raft_consensus.cc:729] T 992544bbea6547de96298fd32d74b232 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:55:17.520094 2247 consensus_queue.cc:260] T 992544bbea6547de96298fd32d74b232 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:17.530588 2247 ts_tablet_manager.cc:1398] T 992544bbea6547de96298fd32d74b232 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240502 01:55:17.536593 25021 tablet_copy_service.cc:342] P 41c70f373c9d49a49891b03a12c6cb2b: Request end of tablet copy session c2a28baa7ee2469294d94b7fe5455cff-992544bbea6547de96298fd32d74b232 received from {username='slave'} at 127.14.212.196:54787
I20240502 01:55:17.536761 25021 tablet_copy_service.cc:433] P 41c70f373c9d49a49891b03a12c6cb2b: ending tablet copy session c2a28baa7ee2469294d94b7fe5455cff-992544bbea6547de96298fd32d74b232 on tablet 992544bbea6547de96298fd32d74b232 with peer c2a28baa7ee2469294d94b7fe5455cff
I20240502 01:55:17.541314 2254 tablet_copy_client.cc:308] T 8a0ae0d8e8784db9988b3469d318f7fb P 28157bc3e678428fb5f93c077d528513: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:44411
I20240502 01:55:17.542228 22010 tablet_copy_service.cc:140] P 06543b05a16f42099bc86db2a0284412: Received BeginTabletCopySession request for tablet 8a0ae0d8e8784db9988b3469d318f7fb from peer 28157bc3e678428fb5f93c077d528513 ({username='slave'} at 127.14.212.198:53247)
I20240502 01:55:17.542425 22010 tablet_copy_service.cc:161] P 06543b05a16f42099bc86db2a0284412: Beginning new tablet copy session on tablet 8a0ae0d8e8784db9988b3469d318f7fb from peer 28157bc3e678428fb5f93c077d528513 at {username='slave'} at 127.14.212.198:53247: session id = 28157bc3e678428fb5f93c077d528513-8a0ae0d8e8784db9988b3469d318f7fb
I20240502 01:55:17.545583 22010 tablet_copy_source_session.cc:205] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:17.547214 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a0ae0d8e8784db9988b3469d318f7fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:17.559525 2254 tablet_copy_client.cc:783] T 8a0ae0d8e8784db9988b3469d318f7fb P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:17.560075 2254 tablet_copy_client.cc:647] T 8a0ae0d8e8784db9988b3469d318f7fb P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:17.570994 2254 tablet_copy_client.cc:523] T 8a0ae0d8e8784db9988b3469d318f7fb P 28157bc3e678428fb5f93c077d528513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:17.574555 2213 ts_tablet_manager.cc:902] T c0877e287cb94abf98723449b9805f80 P d7449bb1df214bf4a5f93e4631312f08: Initiating tablet copy from peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:55:17.581513 2254 tablet_bootstrap.cc:492] T 8a0ae0d8e8784db9988b3469d318f7fb P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:55:17.618820 2213 tablet_copy_client.cc:308] T c0877e287cb94abf98723449b9805f80 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.197:36677
I20240502 01:55:17.620645 24623 tablet_copy_service.cc:140] P 0c64d40b5d7644bd8b29feaa80eab72f: Received BeginTabletCopySession request for tablet c0877e287cb94abf98723449b9805f80 from peer d7449bb1df214bf4a5f93e4631312f08 ({username='slave'} at 127.14.212.199:34671)
I20240502 01:55:17.620860 24623 tablet_copy_service.cc:161] P 0c64d40b5d7644bd8b29feaa80eab72f: Beginning new tablet copy session on tablet c0877e287cb94abf98723449b9805f80 from peer d7449bb1df214bf4a5f93e4631312f08 at {username='slave'} at 127.14.212.199:34671: session id = d7449bb1df214bf4a5f93e4631312f08-c0877e287cb94abf98723449b9805f80
I20240502 01:55:17.624493 24623 tablet_copy_source_session.cc:205] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:17.627218 2213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0877e287cb94abf98723449b9805f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:17.634279 2213 tablet_copy_client.cc:783] T c0877e287cb94abf98723449b9805f80 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:17.634579 2213 tablet_copy_client.cc:647] T c0877e287cb94abf98723449b9805f80 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:17.638710 2213 tablet_copy_client.cc:523] T c0877e287cb94abf98723449b9805f80 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:17.642325 2213 tablet_bootstrap.cc:492] T c0877e287cb94abf98723449b9805f80 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:55:17.651055 2283 ts_tablet_manager.cc:902] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b: Initiating tablet copy from peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:17.656752 2254 tablet_bootstrap.cc:492] T 8a0ae0d8e8784db9988b3469d318f7fb P 28157bc3e678428fb5f93c077d528513: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:17.657343 2254 tablet_bootstrap.cc:492] T 8a0ae0d8e8784db9988b3469d318f7fb P 28157bc3e678428fb5f93c077d528513: Bootstrap complete.
I20240502 01:55:17.657670 2254 ts_tablet_manager.cc:1367] T 8a0ae0d8e8784db9988b3469d318f7fb P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.076s user 0.058s sys 0.002s
I20240502 01:55:17.658115 2213 tablet_bootstrap.cc:492] T c0877e287cb94abf98723449b9805f80 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:17.658567 2213 tablet_bootstrap.cc:492] T c0877e287cb94abf98723449b9805f80 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap complete.
I20240502 01:55:17.658653 2254 raft_consensus.cc:348] T 8a0ae0d8e8784db9988b3469d318f7fb P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:17.658895 2213 ts_tablet_manager.cc:1367] T c0877e287cb94abf98723449b9805f80 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.012s
I20240502 01:55:17.658959 2254 raft_consensus.cc:729] T 8a0ae0d8e8784db9988b3469d318f7fb P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:55:17.659662 2213 raft_consensus.cc:348] T c0877e287cb94abf98723449b9805f80 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:17.659643 2254 consensus_queue.cc:260] T 8a0ae0d8e8784db9988b3469d318f7fb P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:17.659889 2213 raft_consensus.cc:729] T c0877e287cb94abf98723449b9805f80 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:55:17.660424 2254 ts_tablet_manager.cc:1398] T 8a0ae0d8e8784db9988b3469d318f7fb P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:55:17.661211 22010 tablet_copy_service.cc:342] P 06543b05a16f42099bc86db2a0284412: Request end of tablet copy session 28157bc3e678428fb5f93c077d528513-8a0ae0d8e8784db9988b3469d318f7fb received from {username='slave'} at 127.14.212.198:53247
I20240502 01:55:17.661329 22010 tablet_copy_service.cc:433] P 06543b05a16f42099bc86db2a0284412: ending tablet copy session 28157bc3e678428fb5f93c077d528513-8a0ae0d8e8784db9988b3469d318f7fb on tablet 8a0ae0d8e8784db9988b3469d318f7fb with peer 28157bc3e678428fb5f93c077d528513
I20240502 01:55:17.663400 2213 consensus_queue.cc:260] T c0877e287cb94abf98723449b9805f80 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:17.670025 2213 ts_tablet_manager.cc:1398] T c0877e287cb94abf98723449b9805f80 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.011s user 0.002s sys 0.003s
I20240502 01:55:17.670989 24623 tablet_copy_service.cc:342] P 0c64d40b5d7644bd8b29feaa80eab72f: Request end of tablet copy session d7449bb1df214bf4a5f93e4631312f08-c0877e287cb94abf98723449b9805f80 received from {username='slave'} at 127.14.212.199:34671
I20240502 01:55:17.671203 24623 tablet_copy_service.cc:433] P 0c64d40b5d7644bd8b29feaa80eab72f: ending tablet copy session d7449bb1df214bf4a5f93e4631312f08-c0877e287cb94abf98723449b9805f80 on tablet c0877e287cb94abf98723449b9805f80 with peer d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:17.688138 2283 tablet_copy_client.cc:308] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.198:42375
I20240502 01:55:17.688838 24755 tablet_copy_service.cc:140] P 28157bc3e678428fb5f93c077d528513: Received BeginTabletCopySession request for tablet 0cce126a8ba64fb2b40c4ddd03982f19 from peer 41c70f373c9d49a49891b03a12c6cb2b ({username='slave'} at 127.14.212.200:49321)
I20240502 01:55:17.689021 24755 tablet_copy_service.cc:161] P 28157bc3e678428fb5f93c077d528513: Beginning new tablet copy session on tablet 0cce126a8ba64fb2b40c4ddd03982f19 from peer 41c70f373c9d49a49891b03a12c6cb2b at {username='slave'} at 127.14.212.200:49321: session id = 41c70f373c9d49a49891b03a12c6cb2b-0cce126a8ba64fb2b40c4ddd03982f19
I20240502 01:55:17.691632 24755 tablet_copy_source_session.cc:205] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:17.694033 2283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cce126a8ba64fb2b40c4ddd03982f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:17.695737 21985 consensus_queue.cc:237] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:17.700435 2283 tablet_copy_client.cc:783] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:17.700703 2283 tablet_copy_client.cc:647] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:17.705435 2283 tablet_copy_client.cc:523] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:17.710556 2283 tablet_bootstrap.cc:492] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
W20240502 01:55:17.722515 21924 consensus_peers.cc:485] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 -> Peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Couldn't send request to peer c2a28baa7ee2469294d94b7fe5455cff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c81ce3a90f04b818186bb6e43259d13. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:17.722748 24603 raft_consensus.cc:1261] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:17.723681 2127 consensus_queue.cc:1035] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:17.729799 2290 ts_tablet_manager.cc:902] T 5152c0bcae6946059900d52f0d33ac5b P 759257ea7bef415ca698e60d82199d77: Initiating tablet copy from peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:55:17.730305 24999 raft_consensus.cc:1261] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:17.731089 2127 consensus_queue.cc:1035] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:17.736624 2290 tablet_copy_client.cc:308] T 5152c0bcae6946059900d52f0d33ac5b P 759257ea7bef415ca698e60d82199d77: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.197:36677
I20240502 01:55:17.737423 24623 tablet_copy_service.cc:140] P 0c64d40b5d7644bd8b29feaa80eab72f: Received BeginTabletCopySession request for tablet 5152c0bcae6946059900d52f0d33ac5b from peer 759257ea7bef415ca698e60d82199d77 ({username='slave'} at 127.14.212.193:46525)
I20240502 01:55:17.737579 24623 tablet_copy_service.cc:161] P 0c64d40b5d7644bd8b29feaa80eab72f: Beginning new tablet copy session on tablet 5152c0bcae6946059900d52f0d33ac5b from peer 759257ea7bef415ca698e60d82199d77 at {username='slave'} at 127.14.212.193:46525: session id = 759257ea7bef415ca698e60d82199d77-5152c0bcae6946059900d52f0d33ac5b
I20240502 01:55:17.737701 2283 tablet_bootstrap.cc:492] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:17.738140 2283 tablet_bootstrap.cc:492] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap complete.
I20240502 01:55:17.738416 2283 ts_tablet_manager.cc:1367] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.017s
I20240502 01:55:17.739264 2283 raft_consensus.cc:348] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:17.739486 2283 raft_consensus.cc:729] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:55:17.739768 2283 consensus_queue.cc:260] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:17.740463 24623 tablet_copy_source_session.cc:205] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:17.740564 2283 ts_tablet_manager.cc:1398] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:55:17.743435 24755 tablet_copy_service.cc:342] P 28157bc3e678428fb5f93c077d528513: Request end of tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-0cce126a8ba64fb2b40c4ddd03982f19 received from {username='slave'} at 127.14.212.200:49321
I20240502 01:55:17.743532 2290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5152c0bcae6946059900d52f0d33ac5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:17.743573 24755 tablet_copy_service.cc:433] P 28157bc3e678428fb5f93c077d528513: ending tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-0cce126a8ba64fb2b40c4ddd03982f19 on tablet 0cce126a8ba64fb2b40c4ddd03982f19 with peer 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:55:17.748059 2127 raft_consensus.cc:2932] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:17.750350 2290 tablet_copy_client.cc:783] T 5152c0bcae6946059900d52f0d33ac5b P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:17.750576 2290 tablet_copy_client.cc:647] T 5152c0bcae6946059900d52f0d33ac5b P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:17.750320 24999 raft_consensus.cc:2932] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:17.753773 24603 raft_consensus.cc:2932] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:17.760475 21709 catalog_manager.cc:5484] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:17.768780 2290 tablet_copy_client.cc:523] T 5152c0bcae6946059900d52f0d33ac5b P 759257ea7bef415ca698e60d82199d77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:17.772511 2290 tablet_bootstrap.cc:492] T 5152c0bcae6946059900d52f0d33ac5b P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:55:17.777714 2290 tablet_bootstrap.cc:492] T 5152c0bcae6946059900d52f0d33ac5b P 759257ea7bef415ca698e60d82199d77: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:17.778512 2290 tablet_bootstrap.cc:492] T 5152c0bcae6946059900d52f0d33ac5b P 759257ea7bef415ca698e60d82199d77: Bootstrap complete.
I20240502 01:55:17.778856 2290 ts_tablet_manager.cc:1367] T 5152c0bcae6946059900d52f0d33ac5b P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.006s user 0.007s sys 0.002s
I20240502 01:55:17.779592 2290 raft_consensus.cc:348] T 5152c0bcae6946059900d52f0d33ac5b P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:17.779793 2290 raft_consensus.cc:729] T 5152c0bcae6946059900d52f0d33ac5b P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:55:17.780056 2290 consensus_queue.cc:260] T 5152c0bcae6946059900d52f0d33ac5b P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:17.780951 2290 ts_tablet_manager.cc:1398] T 5152c0bcae6946059900d52f0d33ac5b P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:17.781675 24623 tablet_copy_service.cc:342] P 0c64d40b5d7644bd8b29feaa80eab72f: Request end of tablet copy session 759257ea7bef415ca698e60d82199d77-5152c0bcae6946059900d52f0d33ac5b received from {username='slave'} at 127.14.212.193:46525
I20240502 01:55:17.781765 24623 tablet_copy_service.cc:433] P 0c64d40b5d7644bd8b29feaa80eab72f: ending tablet copy session 759257ea7bef415ca698e60d82199d77-5152c0bcae6946059900d52f0d33ac5b on tablet 5152c0bcae6946059900d52f0d33ac5b with peer 759257ea7bef415ca698e60d82199d77
I20240502 01:55:17.806694 24864 consensus_queue.cc:237] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:17.809441 24735 raft_consensus.cc:1261] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:55:17.809549 24801 consensus_peers.cc:485] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2c3475bcad94e3dad4be87376e500bc. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:17.809986 22120 raft_consensus.cc:1261] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:17.810045 2019 consensus_queue.cc:1035] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:17.811663 2019 consensus_queue.cc:1035] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:17.813818 2254 ts_tablet_manager.cc:902] T 3be66f4b90824a21a5f41915c9041970 P 28157bc3e678428fb5f93c077d528513: Initiating tablet copy from peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:55:17.833551 1079 raft_consensus.cc:2932] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:17.834425 24735 raft_consensus.cc:2932] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:17.837642 2254 tablet_copy_client.cc:308] T 3be66f4b90824a21a5f41915c9041970 P 28157bc3e678428fb5f93c077d528513: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.193:33785
I20240502 01:55:17.838557 21878 tablet_copy_service.cc:140] P 759257ea7bef415ca698e60d82199d77: Received BeginTabletCopySession request for tablet 3be66f4b90824a21a5f41915c9041970 from peer 28157bc3e678428fb5f93c077d528513 ({username='slave'} at 127.14.212.198:60613)
I20240502 01:55:17.838707 21878 tablet_copy_service.cc:161] P 759257ea7bef415ca698e60d82199d77: Beginning new tablet copy session on tablet 3be66f4b90824a21a5f41915c9041970 from peer 28157bc3e678428fb5f93c077d528513 at {username='slave'} at 127.14.212.198:60613: session id = 28157bc3e678428fb5f93c077d528513-3be66f4b90824a21a5f41915c9041970
I20240502 01:55:17.841567 22118 raft_consensus.cc:2932] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:17.843122 21878 tablet_copy_source_session.cc:205] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:17.845970 21714 catalog_manager.cc:5484] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 reported cstate change: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:17.847504 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be66f4b90824a21a5f41915c9041970. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:17.854630 2254 tablet_copy_client.cc:783] T 3be66f4b90824a21a5f41915c9041970 P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:17.855038 2254 tablet_copy_client.cc:647] T 3be66f4b90824a21a5f41915c9041970 P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:17.874043 24864 raft_consensus.cc:1203] T b1c6c622638444bd854616902b8d91ad P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:17.879637 2254 tablet_copy_client.cc:523] T 3be66f4b90824a21a5f41915c9041970 P 28157bc3e678428fb5f93c077d528513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:17.883360 2254 tablet_bootstrap.cc:492] T 3be66f4b90824a21a5f41915c9041970 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:55:17.980413 2213 ts_tablet_manager.cc:902] T 3a27dc6af117475cb227e3526d2a9007 P d7449bb1df214bf4a5f93e4631312f08: Initiating tablet copy from peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:55:17.980908 22251 raft_consensus.cc:1203] T 992544bbea6547de96298fd32d74b232 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:55:17.989183 2254 tablet_bootstrap.cc:492] T 3be66f4b90824a21a5f41915c9041970 P 28157bc3e678428fb5f93c077d528513: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:17.989585 2254 tablet_bootstrap.cc:492] T 3be66f4b90824a21a5f41915c9041970 P 28157bc3e678428fb5f93c077d528513: Bootstrap complete.
I20240502 01:55:17.989861 2254 ts_tablet_manager.cc:1367] T 3be66f4b90824a21a5f41915c9041970 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.107s user 0.037s sys 0.020s
I20240502 01:55:17.990651 2254 raft_consensus.cc:348] T 3be66f4b90824a21a5f41915c9041970 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:17.990875 2254 raft_consensus.cc:729] T 3be66f4b90824a21a5f41915c9041970 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:55:17.991149 2254 consensus_queue.cc:260] T 3be66f4b90824a21a5f41915c9041970 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:17.991710 2254 ts_tablet_manager.cc:1398] T 3be66f4b90824a21a5f41915c9041970 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:17.995237 21878 tablet_copy_service.cc:342] P 759257ea7bef415ca698e60d82199d77: Request end of tablet copy session 28157bc3e678428fb5f93c077d528513-3be66f4b90824a21a5f41915c9041970 received from {username='slave'} at 127.14.212.198:60613
I20240502 01:55:17.995370 21878 tablet_copy_service.cc:433] P 759257ea7bef415ca698e60d82199d77: ending tablet copy session 28157bc3e678428fb5f93c077d528513-3be66f4b90824a21a5f41915c9041970 on tablet 3be66f4b90824a21a5f41915c9041970 with peer 28157bc3e678428fb5f93c077d528513
I20240502 01:55:18.012053 2213 tablet_copy_client.cc:308] T 3a27dc6af117475cb227e3526d2a9007 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.196:43093
I20240502 01:55:18.013218 22271 tablet_copy_service.cc:140] P c2a28baa7ee2469294d94b7fe5455cff: Received BeginTabletCopySession request for tablet 3a27dc6af117475cb227e3526d2a9007 from peer d7449bb1df214bf4a5f93e4631312f08 ({username='slave'} at 127.14.212.199:55493)
I20240502 01:55:18.013414 22271 tablet_copy_service.cc:161] P c2a28baa7ee2469294d94b7fe5455cff: Beginning new tablet copy session on tablet 3a27dc6af117475cb227e3526d2a9007 from peer d7449bb1df214bf4a5f93e4631312f08 at {username='slave'} at 127.14.212.199:55493: session id = d7449bb1df214bf4a5f93e4631312f08-3a27dc6af117475cb227e3526d2a9007
I20240502 01:55:18.014618 24867 raft_consensus.cc:1203] T c0877e287cb94abf98723449b9805f80 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:18.016072 22271 tablet_copy_source_session.cc:205] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:18.018381 2213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a27dc6af117475cb227e3526d2a9007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:18.026219 2213 tablet_copy_client.cc:783] T 3a27dc6af117475cb227e3526d2a9007 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:18.026428 2213 tablet_copy_client.cc:647] T 3a27dc6af117475cb227e3526d2a9007 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:18.028147 2213 tablet_copy_client.cc:523] T 3a27dc6af117475cb227e3526d2a9007 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:18.030720 2213 tablet_bootstrap.cc:492] T 3a27dc6af117475cb227e3526d2a9007 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:55:18.031271 24735 raft_consensus.cc:1203] T 8a0ae0d8e8784db9988b3469d318f7fb P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:55:18.038372 21988 consensus_queue.cc:237] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:18.042493 24999 raft_consensus.cc:1261] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:18.043226 24735 raft_consensus.cc:1261] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:18.043390 2128 consensus_queue.cc:1035] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:55:18.044142 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37827d24c61a465f88b679cd694df62f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:18.044519 2128 consensus_queue.cc:1035] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:18.062196 2128 raft_consensus.cc:2932] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:18.063125 24735 raft_consensus.cc:2932] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:18.073699 24999 raft_consensus.cc:2932] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:18.080701 21714 catalog_manager.cc:5484] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513 reported cstate change: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:18.101691 2213 tablet_bootstrap.cc:492] T 3a27dc6af117475cb227e3526d2a9007 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:18.102232 2213 tablet_bootstrap.cc:492] T 3a27dc6af117475cb227e3526d2a9007 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap complete.
I20240502 01:55:18.102594 2213 ts_tablet_manager.cc:1367] T 3a27dc6af117475cb227e3526d2a9007 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.072s user 0.049s sys 0.002s
I20240502 01:55:18.104817 2213 raft_consensus.cc:348] T 3a27dc6af117475cb227e3526d2a9007 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:18.105201 2213 raft_consensus.cc:729] T 3a27dc6af117475cb227e3526d2a9007 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:55:18.105607 2213 consensus_queue.cc:260] T 3a27dc6af117475cb227e3526d2a9007 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:18.107818 2359 ts_tablet_manager.cc:902] T 3c81ce3a90f04b818186bb6e43259d13 P c2a28baa7ee2469294d94b7fe5455cff: Initiating tablet copy from peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:55:18.108503 2213 ts_tablet_manager.cc:1398] T 3a27dc6af117475cb227e3526d2a9007 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 01:55:18.109345 22271 tablet_copy_service.cc:342] P c2a28baa7ee2469294d94b7fe5455cff: Request end of tablet copy session d7449bb1df214bf4a5f93e4631312f08-3a27dc6af117475cb227e3526d2a9007 received from {username='slave'} at 127.14.212.199:55493
I20240502 01:55:18.109453 22271 tablet_copy_service.cc:433] P c2a28baa7ee2469294d94b7fe5455cff: ending tablet copy session d7449bb1df214bf4a5f93e4631312f08-3a27dc6af117475cb227e3526d2a9007 on tablet 3a27dc6af117475cb227e3526d2a9007 with peer d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:18.118789 2358 ts_tablet_manager.cc:902] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f: Initiating tablet copy from peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:55:18.123595 2359 tablet_copy_client.cc:308] T 3c81ce3a90f04b818186bb6e43259d13 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:44411
I20240502 01:55:18.124434 22010 tablet_copy_service.cc:140] P 06543b05a16f42099bc86db2a0284412: Received BeginTabletCopySession request for tablet 3c81ce3a90f04b818186bb6e43259d13 from peer c2a28baa7ee2469294d94b7fe5455cff ({username='slave'} at 127.14.212.196:54573)
I20240502 01:55:18.124601 22010 tablet_copy_service.cc:161] P 06543b05a16f42099bc86db2a0284412: Beginning new tablet copy session on tablet 3c81ce3a90f04b818186bb6e43259d13 from peer c2a28baa7ee2469294d94b7fe5455cff at {username='slave'} at 127.14.212.196:54573: session id = c2a28baa7ee2469294d94b7fe5455cff-3c81ce3a90f04b818186bb6e43259d13
I20240502 01:55:18.127118 22010 tablet_copy_source_session.cc:205] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:18.129413 2359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c81ce3a90f04b818186bb6e43259d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:18.135032 2358 tablet_copy_client.cc:308] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.200:37097
I20240502 01:55:18.135970 2359 tablet_copy_client.cc:783] T 3c81ce3a90f04b818186bb6e43259d13 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:18.135987 25021 tablet_copy_service.cc:140] P 41c70f373c9d49a49891b03a12c6cb2b: Received BeginTabletCopySession request for tablet 469abc1f70fc456b998461bd367dfc84 from peer 0c64d40b5d7644bd8b29feaa80eab72f ({username='slave'} at 127.14.212.197:54097)
I20240502 01:55:18.136195 25021 tablet_copy_service.cc:161] P 41c70f373c9d49a49891b03a12c6cb2b: Beginning new tablet copy session on tablet 469abc1f70fc456b998461bd367dfc84 from peer 0c64d40b5d7644bd8b29feaa80eab72f at {username='slave'} at 127.14.212.197:54097: session id = 0c64d40b5d7644bd8b29feaa80eab72f-469abc1f70fc456b998461bd367dfc84
I20240502 01:55:18.136260 2359 tablet_copy_client.cc:647] T 3c81ce3a90f04b818186bb6e43259d13 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:18.137734 2359 tablet_copy_client.cc:523] T 3c81ce3a90f04b818186bb6e43259d13 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:18.140172 2359 tablet_bootstrap.cc:492] T 3c81ce3a90f04b818186bb6e43259d13 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:55:18.145085 2359 tablet_bootstrap.cc:492] T 3c81ce3a90f04b818186bb6e43259d13 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:18.145383 2359 tablet_bootstrap.cc:492] T 3c81ce3a90f04b818186bb6e43259d13 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap complete.
I20240502 01:55:18.145596 2359 ts_tablet_manager.cc:1367] T 3c81ce3a90f04b818186bb6e43259d13 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.005s user 0.007s sys 0.000s
I20240502 01:55:18.146234 2359 raft_consensus.cc:348] T 3c81ce3a90f04b818186bb6e43259d13 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:18.146420 2359 raft_consensus.cc:729] T 3c81ce3a90f04b818186bb6e43259d13 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:55:18.146677 2359 consensus_queue.cc:260] T 3c81ce3a90f04b818186bb6e43259d13 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:18.147488 2359 ts_tablet_manager.cc:1398] T 3c81ce3a90f04b818186bb6e43259d13 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:18.148169 22010 tablet_copy_service.cc:342] P 06543b05a16f42099bc86db2a0284412: Request end of tablet copy session c2a28baa7ee2469294d94b7fe5455cff-3c81ce3a90f04b818186bb6e43259d13 received from {username='slave'} at 127.14.212.196:54573
I20240502 01:55:18.148280 22010 tablet_copy_service.cc:433] P 06543b05a16f42099bc86db2a0284412: ending tablet copy session c2a28baa7ee2469294d94b7fe5455cff-3c81ce3a90f04b818186bb6e43259d13 on tablet 3c81ce3a90f04b818186bb6e43259d13 with peer c2a28baa7ee2469294d94b7fe5455cff
I20240502 01:55:18.149843 25021 tablet_copy_source_session.cc:205] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:18.151288 2358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 469abc1f70fc456b998461bd367dfc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:18.157236 2358 tablet_copy_client.cc:783] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:18.157486 2358 tablet_copy_client.cc:647] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:18.160137 2358 tablet_copy_client.cc:523] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:18.165385 2358 tablet_bootstrap.cc:492] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:55:18.175213 2358 tablet_bootstrap.cc:492] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:18.175721 2358 tablet_bootstrap.cc:492] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap complete.
I20240502 01:55:18.176054 2358 ts_tablet_manager.cc:1367] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240502 01:55:18.176895 2358 raft_consensus.cc:348] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:18.177139 2358 raft_consensus.cc:729] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:55:18.177410 2358 consensus_queue.cc:260] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:18.178112 2358 ts_tablet_manager.cc:1398] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:55:18.178860 25021 tablet_copy_service.cc:342] P 41c70f373c9d49a49891b03a12c6cb2b: Request end of tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-469abc1f70fc456b998461bd367dfc84 received from {username='slave'} at 127.14.212.197:54097
I20240502 01:55:18.178993 25021 tablet_copy_service.cc:433] P 41c70f373c9d49a49891b03a12c6cb2b: ending tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-469abc1f70fc456b998461bd367dfc84 on tablet 469abc1f70fc456b998461bd367dfc84 with peer 0c64d40b5d7644bd8b29feaa80eab72f
I20240502 01:55:18.197381 24735 consensus_queue.cc:237] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:18.200199 24864 raft_consensus.cc:1261] T 8b377c7402194b42a8eabc747f668fb2 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:18.200412 24999 raft_consensus.cc:1203] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:18.200518 24603 raft_consensus.cc:1261] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:18.200838 2382 consensus_queue.cc:1035] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:55:18.201897 24670 consensus_peers.cc:485] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 -> Peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Couldn't send request to peer c2a28baa7ee2469294d94b7fe5455cff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b377c7402194b42a8eabc747f668fb2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:18.202123 2026 consensus_queue.cc:1035] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:18.208246 2026 raft_consensus.cc:2932] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:18.212258 24864 raft_consensus.cc:2932] T 8b377c7402194b42a8eabc747f668fb2 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:18.212242 24603 raft_consensus.cc:2932] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:18.214651 21709 catalog_manager.cc:5484] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 reported cstate change: config changed from index -1 to 2, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) added. New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:18.272390 21854 raft_consensus.cc:1203] T 5152c0bcae6946059900d52f0d33ac5b P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:18.339815 2409 ts_tablet_manager.cc:902] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77: Initiating tablet copy from peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:55:18.376411 2409 tablet_copy_client.cc:308] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.199:35737
I20240502 01:55:18.377353 24888 tablet_copy_service.cc:140] P d7449bb1df214bf4a5f93e4631312f08: Received BeginTabletCopySession request for tablet a2c3475bcad94e3dad4be87376e500bc from peer 759257ea7bef415ca698e60d82199d77 ({username='slave'} at 127.14.212.193:53739)
I20240502 01:55:18.377571 24888 tablet_copy_service.cc:161] P d7449bb1df214bf4a5f93e4631312f08: Beginning new tablet copy session on tablet a2c3475bcad94e3dad4be87376e500bc from peer 759257ea7bef415ca698e60d82199d77 at {username='slave'} at 127.14.212.193:53739: session id = 759257ea7bef415ca698e60d82199d77-a2c3475bcad94e3dad4be87376e500bc
I20240502 01:55:18.380844 24888 tablet_copy_source_session.cc:205] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:18.382675 2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c3475bcad94e3dad4be87376e500bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:18.389328 2409 tablet_copy_client.cc:783] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:18.389581 2409 tablet_copy_client.cc:647] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:18.393033 2409 tablet_copy_client.cc:523] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:18.396497 2409 tablet_bootstrap.cc:492] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:55:18.400735 24735 raft_consensus.cc:1203] T 3be66f4b90824a21a5f41915c9041970 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:55:18.405442 2409 tablet_bootstrap.cc:492] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:18.406142 2409 tablet_bootstrap.cc:492] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77: Bootstrap complete.
I20240502 01:55:18.406412 2409 ts_tablet_manager.cc:1367] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.003s
I20240502 01:55:18.407138 2409 raft_consensus.cc:348] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:18.407311 2409 raft_consensus.cc:729] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:55:18.407514 2409 consensus_queue.cc:260] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:18.408123 2409 ts_tablet_manager.cc:1398] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:55:18.410888 24867 raft_consensus.cc:1203] T 3a27dc6af117475cb227e3526d2a9007 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:55:18.417341 24888 tablet_copy_service.cc:342] P d7449bb1df214bf4a5f93e4631312f08: Request end of tablet copy session 759257ea7bef415ca698e60d82199d77-a2c3475bcad94e3dad4be87376e500bc received from {username='slave'} at 127.14.212.193:53739
I20240502 01:55:18.417555 24888 tablet_copy_service.cc:433] P d7449bb1df214bf4a5f93e4631312f08: ending tablet copy session 759257ea7bef415ca698e60d82199d77-a2c3475bcad94e3dad4be87376e500bc on tablet a2c3475bcad94e3dad4be87376e500bc with peer 759257ea7bef415ca698e60d82199d77
I20240502 01:55:18.476570 2213 ts_tablet_manager.cc:902] T 37827d24c61a465f88b679cd694df62f P d7449bb1df214bf4a5f93e4631312f08: Initiating tablet copy from peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:55:18.529855 2213 tablet_copy_client.cc:308] T 37827d24c61a465f88b679cd694df62f P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:44411
I20240502 01:55:18.530800 22010 tablet_copy_service.cc:140] P 06543b05a16f42099bc86db2a0284412: Received BeginTabletCopySession request for tablet 37827d24c61a465f88b679cd694df62f from peer d7449bb1df214bf4a5f93e4631312f08 ({username='slave'} at 127.14.212.199:39887)
I20240502 01:55:18.530977 22010 tablet_copy_service.cc:161] P 06543b05a16f42099bc86db2a0284412: Beginning new tablet copy session on tablet 37827d24c61a465f88b679cd694df62f from peer d7449bb1df214bf4a5f93e4631312f08 at {username='slave'} at 127.14.212.199:39887: session id = d7449bb1df214bf4a5f93e4631312f08-37827d24c61a465f88b679cd694df62f
I20240502 01:55:18.532764 22251 raft_consensus.cc:1203] T 3c81ce3a90f04b818186bb6e43259d13 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:18.535996 22010 tablet_copy_source_session.cc:205] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:18.538412 2213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37827d24c61a465f88b679cd694df62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:18.545426 2213 tablet_copy_client.cc:783] T 37827d24c61a465f88b679cd694df62f P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:18.545755 2213 tablet_copy_client.cc:647] T 37827d24c61a465f88b679cd694df62f P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:18.553066 2213 tablet_copy_client.cc:523] T 37827d24c61a465f88b679cd694df62f P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:18.557885 2213 tablet_bootstrap.cc:492] T 37827d24c61a465f88b679cd694df62f P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:55:18.581002 2213 tablet_bootstrap.cc:492] T 37827d24c61a465f88b679cd694df62f P d7449bb1df214bf4a5f93e4631312f08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:18.581521 2213 tablet_bootstrap.cc:492] T 37827d24c61a465f88b679cd694df62f P d7449bb1df214bf4a5f93e4631312f08: Bootstrap complete.
I20240502 01:55:18.582559 2213 ts_tablet_manager.cc:1367] T 37827d24c61a465f88b679cd694df62f P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.025s user 0.018s sys 0.001s
I20240502 01:55:18.583477 2213 raft_consensus.cc:348] T 37827d24c61a465f88b679cd694df62f P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:18.583796 2213 raft_consensus.cc:729] T 37827d24c61a465f88b679cd694df62f P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:55:18.584085 2213 consensus_queue.cc:260] T 37827d24c61a465f88b679cd694df62f P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:18.585127 2213 ts_tablet_manager.cc:1398] T 37827d24c61a465f88b679cd694df62f P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:18.586302 22010 tablet_copy_service.cc:342] P 06543b05a16f42099bc86db2a0284412: Request end of tablet copy session d7449bb1df214bf4a5f93e4631312f08-37827d24c61a465f88b679cd694df62f received from {username='slave'} at 127.14.212.199:39887
I20240502 01:55:18.587385 22010 tablet_copy_service.cc:433] P 06543b05a16f42099bc86db2a0284412: ending tablet copy session d7449bb1df214bf4a5f93e4631312f08-37827d24c61a465f88b679cd694df62f on tablet 37827d24c61a465f88b679cd694df62f with peer d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:18.620698 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:18.644248 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:18.714177 24583 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:18.724738 24599 raft_consensus.cc:1203] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:18.757057 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:18.771122 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:18.791352 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:18.805063 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:18.826175 24981 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:18.829845 2459 ts_tablet_manager.cc:902] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff: Initiating tablet copy from peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:18.891448 24864 raft_consensus.cc:1203] T 37827d24c61a465f88b679cd694df62f P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:18.937172 2459 tablet_copy_client.cc:308] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.198:42375
I20240502 01:55:18.938161 24755 tablet_copy_service.cc:140] P 28157bc3e678428fb5f93c077d528513: Received BeginTabletCopySession request for tablet 8b377c7402194b42a8eabc747f668fb2 from peer c2a28baa7ee2469294d94b7fe5455cff ({username='slave'} at 127.14.212.196:47123)
I20240502 01:55:18.938331 24755 tablet_copy_service.cc:161] P 28157bc3e678428fb5f93c077d528513: Beginning new tablet copy session on tablet 8b377c7402194b42a8eabc747f668fb2 from peer c2a28baa7ee2469294d94b7fe5455cff at {username='slave'} at 127.14.212.196:47123: session id = c2a28baa7ee2469294d94b7fe5455cff-8b377c7402194b42a8eabc747f668fb2
I20240502 01:55:18.943478 21854 raft_consensus.cc:1203] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:18.944583 24755 tablet_copy_source_session.cc:205] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:18.946282 2459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b377c7402194b42a8eabc747f668fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:18.963291 2459 tablet_copy_client.cc:783] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:18.963590 2459 tablet_copy_client.cc:647] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:18.968530 2459 tablet_copy_client.cc:523] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:18.972271 2459 tablet_bootstrap.cc:492] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap starting.
I20240502 01:55:19.011091 2459 tablet_bootstrap.cc:492] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:19.011490 2459 tablet_bootstrap.cc:492] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff: Bootstrap complete.
I20240502 01:55:19.011768 2459 ts_tablet_manager.cc:1367] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.002s
I20240502 01:55:19.012640 2459 raft_consensus.cc:348] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:19.012928 2459 raft_consensus.cc:729] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Initialized, Role: FOLLOWER
I20240502 01:55:19.013245 2459 consensus_queue.cc:260] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:19.013940 2459 ts_tablet_manager.cc:1398] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:55:19.015308 24755 tablet_copy_service.cc:342] P 28157bc3e678428fb5f93c077d528513: Request end of tablet copy session c2a28baa7ee2469294d94b7fe5455cff-8b377c7402194b42a8eabc747f668fb2 received from {username='slave'} at 127.14.212.196:47123
I20240502 01:55:19.015446 24755 tablet_copy_service.cc:433] P 28157bc3e678428fb5f93c077d528513: ending tablet copy session c2a28baa7ee2469294d94b7fe5455cff-8b377c7402194b42a8eabc747f668fb2 on tablet 8b377c7402194b42a8eabc747f668fb2 with peer c2a28baa7ee2469294d94b7fe5455cff
I20240502 01:55:19.269829 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:19.292155 22249 raft_consensus.cc:1203] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:19.329377 24583 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:19.349485 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:19.383103 24981 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:19.386813 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:19.438978 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:19.477305 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:19.526355 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:19.733855 24867 consensus_queue.cc:237] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:19.736094 24735 raft_consensus.cc:1261] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:19.736164 21854 raft_consensus.cc:1261] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:19.736766 1079 consensus_queue.cc:1035] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:19.737551 1079 consensus_queue.cc:1035] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:19.740012 2526 raft_consensus.cc:2932] T a2c3475bcad94e3dad4be87376e500bc P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:19.740820 24735 raft_consensus.cc:2932] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:19.741535 21854 raft_consensus.cc:2932] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:19.746914 21715 catalog_manager.cc:5484] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 reported cstate change: config changed from index 2 to 3, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:19.771754 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet a2c3475bcad94e3dad4be87376e500bc with delete_type TABLET_DATA_TOMBSTONED (TS c64840657cdf4b93980bd34120c94d9a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40366
I20240502 01:55:19.772814 2559 tablet_replica.cc:330] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:55:19.773097 2559 raft_consensus.cc:2229] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:19.773923 2559 raft_consensus.cc:2258] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:19.774997 2559 ts_tablet_manager.cc:1875] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:19.780807 2559 ts_tablet_manager.cc:1888] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:19.780980 2559 log.cc:1196] T a2c3475bcad94e3dad4be87376e500bc P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/a2c3475bcad94e3dad4be87376e500bc
I20240502 01:55:19.782505 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet a2c3475bcad94e3dad4be87376e500bc (table rebalancer_extra_table_28 [id=3bb44c54e77145c39bccee47a8077319]) successfully deleted
I20240502 01:55:20.111122 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:20.114436 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:20.149194 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:20.153394 24981 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:20.157143 24583 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:20.221148 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:20.262384 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:20.288220 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:20.408424 21988 consensus_queue.cc:237] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:20.410601 22249 raft_consensus.cc:1261] T 3c81ce3a90f04b818186bb6e43259d13 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:20.411005 24601 raft_consensus.cc:1261] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:20.411399 2128 consensus_queue.cc:1035] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:20.411669 2351 consensus_queue.cc:1035] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:20.415202 2536 raft_consensus.cc:2932] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:20.415809 22251 raft_consensus.cc:2932] T 3c81ce3a90f04b818186bb6e43259d13 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:20.415858 24601 raft_consensus.cc:2932] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:20.424927 21715 catalog_manager.cc:5484] T 3c81ce3a90f04b818186bb6e43259d13 P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index 2 to 3, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:20.439242 24981 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c81ce3a90f04b818186bb6e43259d13 with delete_type TABLET_DATA_TOMBSTONED (TS 41c70f373c9d49a49891b03a12c6cb2b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:39964
I20240502 01:55:20.440827 2648 tablet_replica.cc:330] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:55:20.441143 2648 raft_consensus.cc:2229] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:20.442112 2648 raft_consensus.cc:2258] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:20.444298 2648 ts_tablet_manager.cc:1875] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:20.450920 2648 ts_tablet_manager.cc:1888] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:20.451131 2648 log.cc:1196] T 3c81ce3a90f04b818186bb6e43259d13 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/3c81ce3a90f04b818186bb6e43259d13
I20240502 01:55:20.451983 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet 3c81ce3a90f04b818186bb6e43259d13 (table rebalancer_extra_table_48 [id=ca142102a7e44673b2cac39cee9b5bcd]) successfully deleted
I20240502 01:55:20.759358 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:20.793478 24583 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:20.802346 24981 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:20.836591 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:20.850940 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:20.859467 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:20.910598 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:20.945724 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:21.096705 25000 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "469abc1f70fc456b998461bd367dfc84"
dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40232
I20240502 01:55:21.096980 25000 raft_consensus.cc:595] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:21.135449 2701 raft_consensus.cc:979] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b: : Instructing follower 0c64d40b5d7644bd8b29feaa80eab72f to start an election
I20240502 01:55:21.135658 2701 raft_consensus.cc:1067] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Signalling peer 0c64d40b5d7644bd8b29feaa80eab72f to start an election
I20240502 01:55:21.136482 24599 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "469abc1f70fc456b998461bd367dfc84"
dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
from {username='slave'} at 127.14.212.200:39881
I20240502 01:55:21.136691 24599 raft_consensus.cc:482] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:55:21.136797 24599 raft_consensus.cc:3037] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:21.139577 24599 raft_consensus.cc:504] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:21.140965 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "469abc1f70fc456b998461bd367dfc84" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:55:21.141177 21854 raft_consensus.cc:3037] T 469abc1f70fc456b998461bd367dfc84 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:21.141683 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "469abc1f70fc456b998461bd367dfc84" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:55:21.141901 22251 raft_consensus.cc:3037] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:21.143478 21854 raft_consensus.cc:2454] T 469abc1f70fc456b998461bd367dfc84 P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 2.
I20240502 01:55:21.144277 22251 raft_consensus.cc:2454] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 2.
I20240502 01:55:21.144387 24599 leader_election.cc:288] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:55:21.144698 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "469abc1f70fc456b998461bd367dfc84" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:55:21.144951 24999 raft_consensus.cc:3032] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:55:21.145043 24999 raft_consensus.cc:729] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:55:21.145349 24538 leader_election.cc:302] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:55:21.145443 24999 consensus_queue.cc:260] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:21.145696 2689 raft_consensus.cc:2781] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:21.145866 2689 raft_consensus.cc:686] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:55:21.145915 24999 raft_consensus.cc:3037] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:21.146245 2689 consensus_queue.cc:237] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:21.148519 24999 raft_consensus.cc:2454] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 2.
I20240502 01:55:21.162784 21710 catalog_manager.cc:5484] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 1 to 2, leader changed from 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 2 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:21.380045 24583 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:21.395226 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:21.446018 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:21.472569 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:21.573402 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:21.582495 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:21.593125 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:21.659708 22249 raft_consensus.cc:1261] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:55:21.660284 2687 consensus_queue.cc:1035] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:21.670328 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:21.711354 21854 raft_consensus.cc:1261] T 469abc1f70fc456b998461bd367dfc84 P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:55:21.712029 2689 consensus_queue.cc:1035] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:21.744230 25000 raft_consensus.cc:1261] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:55:21.744792 2689 consensus_queue.cc:1035] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:22.138028 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:22.142580 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:22.159564 24583 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:22.188315 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:22.189348 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:22.284646 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:22.347095 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:22.361847 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:22.498790 21854 consensus_queue.cc:237] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:22.501314 22249 raft_consensus.cc:1261] T 3be66f4b90824a21a5f41915c9041970 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:22.502300 24735 raft_consensus.cc:1261] T 3be66f4b90824a21a5f41915c9041970 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:22.503044 2457 consensus_queue.cc:1035] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:22.504981 2457 consensus_queue.cc:1035] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:22.507160 2457 raft_consensus.cc:2932] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:22.513624 24735 raft_consensus.cc:2932] T 3be66f4b90824a21a5f41915c9041970 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:22.522378 21713 catalog_manager.cc:5484] T 3be66f4b90824a21a5f41915c9041970 P 28157bc3e678428fb5f93c077d528513 reported cstate change: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:22.551541 22251 raft_consensus.cc:2932] T 3be66f4b90824a21a5f41915c9041970 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:22.552268 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be66f4b90824a21a5f41915c9041970 with delete_type TABLET_DATA_TOMBSTONED (TS c64840657cdf4b93980bd34120c94d9a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40366
I20240502 01:55:22.561095 2876 tablet_replica.cc:330] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:55:22.561430 2876 raft_consensus.cc:2229] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:22.561800 2876 raft_consensus.cc:2258] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:22.562646 2876 ts_tablet_manager.cc:1875] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:22.568504 2876 ts_tablet_manager.cc:1888] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:55:22.568708 2876 log.cc:1196] T 3be66f4b90824a21a5f41915c9041970 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/3be66f4b90824a21a5f41915c9041970
I20240502 01:55:22.569618 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 3be66f4b90824a21a5f41915c9041970 (table rebalance_test_table_13 [id=b462b431b51b4d45baf0ff5a495aa90a]) successfully deleted
I20240502 01:55:22.873797 24583 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:22.896399 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:23.006364 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:23.031404 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:23.041334 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:23.079167 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:23.091552 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:23.098004 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:23.221705 24599 consensus_queue.cc:237] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:23.223740 24735 raft_consensus.cc:1261] T 5152c0bcae6946059900d52f0d33ac5b P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:23.223815 21854 raft_consensus.cc:1261] T 5152c0bcae6946059900d52f0d33ac5b P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:23.224267 2687 consensus_queue.cc:1035] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:23.224597 2689 consensus_queue.cc:1035] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:23.228174 2687 raft_consensus.cc:2932] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:23.230003 24735 raft_consensus.cc:2932] T 5152c0bcae6946059900d52f0d33ac5b P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:23.232553 21856 raft_consensus.cc:2932] T 5152c0bcae6946059900d52f0d33ac5b P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:23.236083 21711 catalog_manager.cc:5484] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: config changed from index 2 to 3, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:23.247447 22229 tablet_service.cc:1514] Processing DeleteTablet for tablet 5152c0bcae6946059900d52f0d33ac5b with delete_type TABLET_DATA_TOMBSTONED (TS c2a28baa7ee2469294d94b7fe5455cff not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50898
I20240502 01:55:23.248348 2961 tablet_replica.cc:330] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:55:23.248657 2961 raft_consensus.cc:2229] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:23.249048 2961 raft_consensus.cc:2258] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:23.249811 2961 ts_tablet_manager.cc:1875] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:23.256403 2961 ts_tablet_manager.cc:1888] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:23.256546 2961 log.cc:1196] T 5152c0bcae6946059900d52f0d33ac5b P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/5152c0bcae6946059900d52f0d33ac5b
I20240502 01:55:23.257481 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 5152c0bcae6946059900d52f0d33ac5b (table rebalancer_extra_table_14 [id=9f45904cc4cd41f0962b57ad839bb4c6]) successfully deleted
I20240502 01:55:23.596050 24583 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:23.603044 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:23.622931 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:23.633168 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:23.653740 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:23.699476 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:23.736812 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:23.739679 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:24.060012 24733 consensus_queue.cc:237] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:24.063288 25000 raft_consensus.cc:1261] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:24.063905 22118 raft_consensus.cc:1261] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:24.064525 2706 consensus_queue.cc:1035] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:24.065070 2765 consensus_queue.cc:1035] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:24.071815 2765 raft_consensus.cc:2932] T 0cce126a8ba64fb2b40c4ddd03982f19 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:24.072494 22118 raft_consensus.cc:2932] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:24.083029 21710 catalog_manager.cc:5484] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:24.108907 25000 raft_consensus.cc:2932] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:24.117264 24583 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cce126a8ba64fb2b40c4ddd03982f19 with delete_type TABLET_DATA_TOMBSTONED (TS 0c64d40b5d7644bd8b29feaa80eab72f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53832
I20240502 01:55:24.129714 3048 tablet_replica.cc:330] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:55:24.130095 3048 raft_consensus.cc:2229] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:24.130389 3048 raft_consensus.cc:2258] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:24.131315 3048 ts_tablet_manager.cc:1875] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:24.141638 3048 ts_tablet_manager.cc:1888] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:24.142227 3048 log.cc:1196] T 0cce126a8ba64fb2b40c4ddd03982f19 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/0cce126a8ba64fb2b40c4ddd03982f19
I20240502 01:55:24.145224 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet 0cce126a8ba64fb2b40c4ddd03982f19 (table rebalancer_extra_table_0 [id=ee5b662f59f047049d4a2a52984c0cfd]) successfully deleted
I20240502 01:55:24.449041 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:24.466532 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:24.467576 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:24.471683 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:24.493985 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:24.518311 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:24.538519 24583 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:24.649544 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:24.771993 24599 consensus_queue.cc:237] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:24.774057 22251 raft_consensus.cc:1261] T c0877e287cb94abf98723449b9805f80 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:24.774057 24864 raft_consensus.cc:1261] T c0877e287cb94abf98723449b9805f80 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:24.774714 3110 consensus_queue.cc:1035] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:24.775022 2687 consensus_queue.cc:1035] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:24.777882 3130 raft_consensus.cc:2932] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:24.778698 22251 raft_consensus.cc:2932] T c0877e287cb94abf98723449b9805f80 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:24.779024 24864 raft_consensus.cc:2932] T c0877e287cb94abf98723449b9805f80 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:24.784976 21708 catalog_manager.cc:5484] T c0877e287cb94abf98723449b9805f80 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:24.793726 21837 tablet_service.cc:1514] Processing DeleteTablet for tablet c0877e287cb94abf98723449b9805f80 with delete_type TABLET_DATA_TOMBSTONED (TS 759257ea7bef415ca698e60d82199d77 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47656
I20240502 01:55:24.806074 3134 tablet_replica.cc:330] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:55:24.806500 3134 raft_consensus.cc:2229] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:24.806867 3134 raft_consensus.cc:2258] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:24.807729 3134 ts_tablet_manager.cc:1875] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:24.814097 3134 ts_tablet_manager.cc:1888] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:24.814250 3134 log.cc:1196] T c0877e287cb94abf98723449b9805f80 P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/c0877e287cb94abf98723449b9805f80
I20240502 01:55:24.815111 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet c0877e287cb94abf98723449b9805f80 (table rebalancer_extra_table_44 [id=75c3f22aa26848b68c2a3c2b5a6a5b3d]) successfully deleted
I20240502 01:55:25.117477 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:25.130285 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:25.222539 24583 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:25.228405 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:25.260363 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:25.279305 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:25.300173 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 01:55:25.345343 24668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.194:44411, user_credentials={real_user=slave}} blocked reactor thread for 81154.4us
I20240502 01:55:25.364266 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 01:55:25.426259 24936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.195:38879, user_credentials={real_user=slave}} blocked reactor thread for 61631.3us
I20240502 01:55:25.473963 21704 maintenance_manager.cc:419] P f9e4d9a6ca2a4b698f272d98462e62bb: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033388
I20240502 01:55:25.595278 21703 maintenance_manager.cc:643] P f9e4d9a6ca2a4b698f272d98462e62bb: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.121s user 0.053s sys 0.024s Metrics: {"bytes_written":91028,"cfile_init":1,"dirs.queue_time_us":402,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":928,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2088,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":247,"thread_start_us":283,"threads_started":1}
I20240502 01:55:25.597095 21704 maintenance_manager.cc:419] P f9e4d9a6ca2a4b698f272d98462e62bb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 29346 bytes on disk
I20240502 01:55:25.598076 21703 maintenance_manager.cc:643] P f9e4d9a6ca2a4b698f272d98462e62bb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240502 01:55:25.606324 21988 consensus_queue.cc:237] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:25.609855 24735 raft_consensus.cc:1261] T 8a0ae0d8e8784db9988b3469d318f7fb P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:25.609910 21856 raft_consensus.cc:1261] T 8a0ae0d8e8784db9988b3469d318f7fb P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:25.610469 3106 consensus_queue.cc:1035] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:25.610782 3197 consensus_queue.cc:1035] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:25.613660 3106 raft_consensus.cc:2932] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:25.614537 24735 raft_consensus.cc:2932] T 8a0ae0d8e8784db9988b3469d318f7fb P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:25.617628 21854 raft_consensus.cc:2932] T 8a0ae0d8e8784db9988b3469d318f7fb P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:25.621809 21706 catalog_manager.cc:5484] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:25.640205 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a0ae0d8e8784db9988b3469d318f7fb with delete_type TABLET_DATA_TOMBSTONED (TS c64840657cdf4b93980bd34120c94d9a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40366
I20240502 01:55:25.641028 3226 tablet_replica.cc:330] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:55:25.641229 3226 raft_consensus.cc:2229] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:25.641413 3226 raft_consensus.cc:2258] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:25.642335 3226 ts_tablet_manager.cc:1875] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:25.649636 3226 ts_tablet_manager.cc:1888] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:55:25.649849 3226 log.cc:1196] T 8a0ae0d8e8784db9988b3469d318f7fb P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/8a0ae0d8e8784db9988b3469d318f7fb
I20240502 01:55:25.650720 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 8a0ae0d8e8784db9988b3469d318f7fb (table rebalance_test_table_11 [id=e67a4afb37114119a9d514153a1563bb]) successfully deleted
I20240502 01:55:26.039991 21969 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:26.084157 24583 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:26.087249 24981 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:26.114173 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:26.196591 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:26.208349 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:26.251981 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:26.322827 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:26.413072 25000 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "992544bbea6547de96298fd32d74b232"
dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40316
I20240502 01:55:26.413280 25000 raft_consensus.cc:595] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:26.474779 2924 raft_consensus.cc:979] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b: : Instructing follower 06543b05a16f42099bc86db2a0284412 to start an election
I20240502 01:55:26.475206 2924 raft_consensus.cc:1067] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Signalling peer 06543b05a16f42099bc86db2a0284412 to start an election
I20240502 01:55:26.476258 21988 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "992544bbea6547de96298fd32d74b232"
dest_uuid: "06543b05a16f42099bc86db2a0284412"
from {username='slave'} at 127.14.212.200:60949
I20240502 01:55:26.476478 21988 raft_consensus.cc:482] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:55:26.476569 21988 raft_consensus.cc:3037] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:26.481985 21988 raft_consensus.cc:504] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:26.483203 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992544bbea6547de96298fd32d74b232" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:55:26.483448 25001 raft_consensus.cc:3032] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:55:26.483542 25001 raft_consensus.cc:729] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:55:26.483537 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992544bbea6547de96298fd32d74b232" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:55:26.483757 22249 raft_consensus.cc:3037] T 992544bbea6547de96298fd32d74b232 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:26.483834 25001 consensus_queue.cc:260] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:26.484143 25001 raft_consensus.cc:3037] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:26.486230 25001 raft_consensus.cc:2454] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 2.
I20240502 01:55:26.486593 22249 raft_consensus.cc:2454] T 992544bbea6547de96298fd32d74b232 P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 2.
I20240502 01:55:26.487911 21924 leader_election.cc:302] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 41c70f373c9d49a49891b03a12c6cb2b, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:55:26.488201 3106 raft_consensus.cc:2781] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:26.488320 3106 raft_consensus.cc:686] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [term 2 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:55:26.488448 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992544bbea6547de96298fd32d74b232" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:55:26.488665 21854 raft_consensus.cc:3037] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:26.488567 3106 consensus_queue.cc:237] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:26.491480 21854 raft_consensus.cc:2454] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 2.
I20240502 01:55:26.494400 21988 leader_election.cc:288] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:55:26.496687 21714 catalog_manager.cc:5484] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 1 to 2, leader changed from 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 2 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:26.718407 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:26.749042 24583 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:26.757138 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:26.793256 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:26.809471 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:26.862208 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:26.874832 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:26.891837 25001 raft_consensus.cc:1261] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:55:26.895072 3197 consensus_queue.cc:1035] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.901512 21854 raft_consensus.cc:1261] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:55:26.902794 3313 consensus_queue.cc:1035] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.915246 22249 raft_consensus.cc:1261] T 992544bbea6547de96298fd32d74b232 P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:55:26.916747 3313 consensus_queue.cc:1035] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.942778 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:27.339479 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:27.397456 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:27.405934 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:27.409832 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:27.413511 24583 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:27.423624 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:27.455914 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:27.539904 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:27.691051 21857 consensus_queue.cc:237] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:27.698622 24866 raft_consensus.cc:1261] T b1c6c622638444bd854616902b8d91ad P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:27.699416 2944 consensus_queue.cc:1035] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:27.700017 25001 raft_consensus.cc:1261] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:27.700625 2944 consensus_queue.cc:1035] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:27.702895 3109 raft_consensus.cc:2932] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:27.704762 25001 raft_consensus.cc:2932] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:27.705504 24866 raft_consensus.cc:2932] T b1c6c622638444bd854616902b8d91ad P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:27.711702 21706 catalog_manager.cc:5484] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:27.732183 24583 tablet_service.cc:1514] Processing DeleteTablet for tablet b1c6c622638444bd854616902b8d91ad with delete_type TABLET_DATA_TOMBSTONED (TS 0c64d40b5d7644bd8b29feaa80eab72f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53832
I20240502 01:55:27.732956 3452 tablet_replica.cc:330] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:55:27.733922 3452 raft_consensus.cc:2229] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:27.734192 3452 raft_consensus.cc:2258] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:27.734895 3452 ts_tablet_manager.cc:1875] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:27.741354 3452 ts_tablet_manager.cc:1888] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:27.741638 3452 log.cc:1196] T b1c6c622638444bd854616902b8d91ad P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/b1c6c622638444bd854616902b8d91ad
I20240502 01:55:27.742594 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet b1c6c622638444bd854616902b8d91ad (table rebalancer_extra_table_2 [id=06ae89337ff143b5a107c994cbecc25a]) successfully deleted
I20240502 01:55:28.107810 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:28.134999 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:28.152719 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:28.173216 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:28.194519 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 01:55:28.217605 22054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.198:42375, user_credentials={real_user=slave}} blocked reactor thread for 100094us
I20240502 01:55:28.287338 24583 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:28.320099 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:28.324982 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:28.466760 22248 consensus_queue.cc:237] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:28.469067 24866 raft_consensus.cc:1261] T 3a27dc6af117475cb227e3526d2a9007 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:28.469067 22118 raft_consensus.cc:1261] T 3a27dc6af117475cb227e3526d2a9007 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:28.469686 3523 consensus_queue.cc:1035] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:28.469971 3189 consensus_queue.cc:1035] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:28.475567 3523 raft_consensus.cc:2932] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:28.476258 22118 raft_consensus.cc:2932] T 3a27dc6af117475cb227e3526d2a9007 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:28.485738 21706 catalog_manager.cc:5484] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:28.493234 24867 raft_consensus.cc:2932] T 3a27dc6af117475cb227e3526d2a9007 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:28.550541 21969 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a27dc6af117475cb227e3526d2a9007 with delete_type TABLET_DATA_TOMBSTONED (TS 06543b05a16f42099bc86db2a0284412 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46720
I20240502 01:55:28.557550 3551 tablet_replica.cc:330] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:55:28.557883 3551 raft_consensus.cc:2229] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:28.558202 3551 raft_consensus.cc:2258] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:28.559065 3551 ts_tablet_manager.cc:1875] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:28.571786 3551 ts_tablet_manager.cc:1888] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:55:28.572139 3551 log.cc:1196] T 3a27dc6af117475cb227e3526d2a9007 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/3a27dc6af117475cb227e3526d2a9007
I20240502 01:55:28.573449 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 3a27dc6af117475cb227e3526d2a9007 (table rebalance_test_table_5 [id=868965c86d0d413fbbeaca41748259c6]) successfully deleted
I20240502 01:55:28.831039 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:28.906394 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:28.910501 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:28.931978 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:28.972105 24583 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:29.000048 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:29.028255 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:29.042069 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:29.194572 22120 consensus_queue.cc:237] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:29.196946 24733 raft_consensus.cc:1261] T daaadb773202454894890c3379c4a58d P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:29.197134 25001 raft_consensus.cc:1261] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:29.197585 3515 consensus_queue.cc:1035] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:29.197875 3513 consensus_queue.cc:1035] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:29.200814 3515 raft_consensus.cc:2932] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:29.205228 24735 raft_consensus.cc:2932] T daaadb773202454894890c3379c4a58d P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:29.210301 21714 catalog_manager.cc:5484] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index 6 to 7, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:29.218999 25001 raft_consensus.cc:2932] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:29.237847 21837 tablet_service.cc:1514] Processing DeleteTablet for tablet daaadb773202454894890c3379c4a58d with delete_type TABLET_DATA_TOMBSTONED (TS 759257ea7bef415ca698e60d82199d77 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47656
I20240502 01:55:29.238526 3638 tablet_replica.cc:330] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:55:29.238746 3638 raft_consensus.cc:2229] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:29.239081 3638 raft_consensus.cc:2258] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:29.240352 3638 ts_tablet_manager.cc:1875] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:29.247079 3638 ts_tablet_manager.cc:1888] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:29.247243 3638 log.cc:1196] T daaadb773202454894890c3379c4a58d P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/daaadb773202454894890c3379c4a58d
I20240502 01:55:29.248278 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet daaadb773202454894890c3379c4a58d (table rebalance_test_table_15 [id=855af8637536470bb3f01aeac50e07e8]) successfully deleted
I20240502 01:55:29.634115 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:29.648525 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:29.650730 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:29.659457 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:29.661295 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:29.676074 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:29.695549 24583 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:29.744235 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:29.963521 24733 consensus_queue.cc:237] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:29.965775 24867 raft_consensus.cc:1261] T 8b377c7402194b42a8eabc747f668fb2 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:29.966321 22249 raft_consensus.cc:1261] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:29.966344 3287 consensus_queue.cc:1035] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:29.966940 3287 consensus_queue.cc:1035] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:29.974396 3690 raft_consensus.cc:2932] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:29.975787 24867 raft_consensus.cc:2932] T 8b377c7402194b42a8eabc747f668fb2 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:29.980165 22249 raft_consensus.cc:2932] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:29.983055 21710 catalog_manager.cc:5484] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 reported cstate change: config changed from index 2 to 3, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) evicted. New cstate: current_term: 1 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:29.999548 24583 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b377c7402194b42a8eabc747f668fb2 with delete_type TABLET_DATA_TOMBSTONED (TS 0c64d40b5d7644bd8b29feaa80eab72f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53832
I20240502 01:55:30.000654 3739 tablet_replica.cc:330] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f: stopping tablet replica
I20240502 01:55:30.000991 3739 raft_consensus.cc:2229] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:30.001242 3739 raft_consensus.cc:2258] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:30.002053 3739 ts_tablet_manager.cc:1875] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:30.008379 3739 ts_tablet_manager.cc:1888] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:30.008513 3739 log.cc:1196] T 8b377c7402194b42a8eabc747f668fb2 P 0c64d40b5d7644bd8b29feaa80eab72f: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/wals/8b377c7402194b42a8eabc747f668fb2
I20240502 01:55:30.009264 21700 catalog_manager.cc:4830] TS 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): tablet 8b377c7402194b42a8eabc747f668fb2 (table rebalancer_extra_table_16 [id=382119de47e04360a975948cedb418bc]) successfully deleted
I20240502 01:55:30.068960 22120 consensus_queue.cc:237] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:30.072064 22249 raft_consensus.cc:1261] T 4a3f86de558243eea0664d120c62f182 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:55:30.072616 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a3f86de558243eea0664d120c62f182. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:30.072798 21989 raft_consensus.cc:1261] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:30.072991 3513 consensus_queue.cc:1035] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:30.073716 3513 consensus_queue.cc:1035] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:30.076663 3513 raft_consensus.cc:2932] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:30.079770 21989 raft_consensus.cc:2932] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:30.080260 22249 raft_consensus.cc:2932] T 4a3f86de558243eea0664d120c62f182 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:30.087963 21714 catalog_manager.cc:5484] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index -1 to 2, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:30.124572 22249 consensus_queue.cc:237] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:30.127818 24599 raft_consensus.cc:1261] T ba18c000cd3f42978ed8fd01a4ded60c P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:30.127964 22120 raft_consensus.cc:1261] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:30.128767 3523 consensus_queue.cc:1035] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:55:30.129513 22184 consensus_peers.cc:485] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba18c000cd3f42978ed8fd01a4ded60c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:30.129716 3523 consensus_queue.cc:1035] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:30.145224 3711 raft_consensus.cc:2932] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:30.150760 22120 raft_consensus.cc:2932] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:30.152841 21710 catalog_manager.cc:5484] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:30.151229 24599 raft_consensus.cc:2932] T ba18c000cd3f42978ed8fd01a4ded60c P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:30.220021 24867 consensus_queue.cc:237] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:30.222589 24733 raft_consensus.cc:1261] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:30.222669 22249 raft_consensus.cc:1261] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:55:30.222991 24804 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 -> Peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): Couldn't send request to peer 41c70f373c9d49a49891b03a12c6cb2b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 458b19e5186f4ce79f618a10e101c605. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:30.223310 3715 consensus_queue.cc:1035] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:30.223610 3492 consensus_queue.cc:1035] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:30.228029 3715 raft_consensus.cc:2932] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:30.228770 22249 raft_consensus.cc:2932] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:30.232650 24733 raft_consensus.cc:2932] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:30.243784 21711 catalog_manager.cc:5484] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: config changed from index -1 to 2, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) added. New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:30.282721 22120 consensus_queue.cc:237] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:30.285094 21989 raft_consensus.cc:1261] T 88457f21ad4a4e00bb372479c13963ec P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:30.285558 22054 consensus_peers.cc:485] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a -> Peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Couldn't send request to peer 0c64d40b5d7644bd8b29feaa80eab72f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88457f21ad4a4e00bb372479c13963ec. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:30.285939 22249 raft_consensus.cc:1261] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:30.286110 3515 consensus_queue.cc:1035] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:30.286545 3513 consensus_queue.cc:1035] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:30.290932 3515 raft_consensus.cc:2932] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:30.293562 21989 raft_consensus.cc:2932] T 88457f21ad4a4e00bb372479c13963ec P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:30.295238 22249 raft_consensus.cc:2932] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:30.303700 21710 catalog_manager.cc:5484] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:30.648151 24583 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:30.657318 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:30.704859 3876 ts_tablet_manager.cc:902] T 4a3f86de558243eea0664d120c62f182 P d7449bb1df214bf4a5f93e4631312f08: Initiating tablet copy from peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:55:30.704826 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:30.755039 3885 ts_tablet_manager.cc:902] T ba18c000cd3f42978ed8fd01a4ded60c P 759257ea7bef415ca698e60d82199d77: Initiating tablet copy from peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:55:30.767334 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:30.769680 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:30.775993 3890 ts_tablet_manager.cc:902] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b: Initiating tablet copy from peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:55:30.776232 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:30.787258 3890 tablet_copy_client.cc:308] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.199:35737
I20240502 01:55:30.791934 24888 tablet_copy_service.cc:140] P d7449bb1df214bf4a5f93e4631312f08: Received BeginTabletCopySession request for tablet 458b19e5186f4ce79f618a10e101c605 from peer 41c70f373c9d49a49891b03a12c6cb2b ({username='slave'} at 127.14.212.200:33739)
I20240502 01:55:30.792100 24888 tablet_copy_service.cc:161] P d7449bb1df214bf4a5f93e4631312f08: Beginning new tablet copy session on tablet 458b19e5186f4ce79f618a10e101c605 from peer 41c70f373c9d49a49891b03a12c6cb2b at {username='slave'} at 127.14.212.200:33739: session id = 41c70f373c9d49a49891b03a12c6cb2b-458b19e5186f4ce79f618a10e101c605
I20240502 01:55:30.795001 24888 tablet_copy_source_session.cc:205] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:30.797561 3890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458b19e5186f4ce79f618a10e101c605. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.822643 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:30.834841 3876 tablet_copy_client.cc:308] T 4a3f86de558243eea0664d120c62f182 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.195:38879
I20240502 01:55:30.836213 22140 tablet_copy_service.cc:140] P c64840657cdf4b93980bd34120c94d9a: Received BeginTabletCopySession request for tablet 4a3f86de558243eea0664d120c62f182 from peer d7449bb1df214bf4a5f93e4631312f08 ({username='slave'} at 127.14.212.199:47983)
I20240502 01:55:30.836395 22140 tablet_copy_service.cc:161] P c64840657cdf4b93980bd34120c94d9a: Beginning new tablet copy session on tablet 4a3f86de558243eea0664d120c62f182 from peer d7449bb1df214bf4a5f93e4631312f08 at {username='slave'} at 127.14.212.199:47983: session id = d7449bb1df214bf4a5f93e4631312f08-4a3f86de558243eea0664d120c62f182
I20240502 01:55:30.839354 22140 tablet_copy_source_session.cc:205] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:30.841388 3876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a3f86de558243eea0664d120c62f182. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.843755 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:30.847908 3890 tablet_copy_client.cc:783] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:30.848269 3890 tablet_copy_client.cc:647] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:30.850390 3890 tablet_copy_client.cc:523] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:30.850847 3876 tablet_copy_client.cc:783] T 4a3f86de558243eea0664d120c62f182 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:30.851130 3876 tablet_copy_client.cc:647] T 4a3f86de558243eea0664d120c62f182 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:30.858618 3885 tablet_copy_client.cc:308] T ba18c000cd3f42978ed8fd01a4ded60c P 759257ea7bef415ca698e60d82199d77: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.196:43093
I20240502 01:55:30.859293 22271 tablet_copy_service.cc:140] P c2a28baa7ee2469294d94b7fe5455cff: Received BeginTabletCopySession request for tablet ba18c000cd3f42978ed8fd01a4ded60c from peer 759257ea7bef415ca698e60d82199d77 ({username='slave'} at 127.14.212.193:44431)
I20240502 01:55:30.859428 22271 tablet_copy_service.cc:161] P c2a28baa7ee2469294d94b7fe5455cff: Beginning new tablet copy session on tablet ba18c000cd3f42978ed8fd01a4ded60c from peer 759257ea7bef415ca698e60d82199d77 at {username='slave'} at 127.14.212.193:44431: session id = 759257ea7bef415ca698e60d82199d77-ba18c000cd3f42978ed8fd01a4ded60c
I20240502 01:55:30.861996 22271 tablet_copy_source_session.cc:205] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:30.865943 3876 tablet_copy_client.cc:523] T 4a3f86de558243eea0664d120c62f182 P d7449bb1df214bf4a5f93e4631312f08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:30.870882 3876 tablet_bootstrap.cc:492] T 4a3f86de558243eea0664d120c62f182 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap starting.
I20240502 01:55:30.875448 3885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba18c000cd3f42978ed8fd01a4ded60c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.878158 3890 tablet_bootstrap.cc:492] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap starting.
I20240502 01:55:30.885159 3897 ts_tablet_manager.cc:902] T 88457f21ad4a4e00bb372479c13963ec P 0c64d40b5d7644bd8b29feaa80eab72f: Initiating tablet copy from peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:55:30.903396 3876 tablet_bootstrap.cc:492] T 4a3f86de558243eea0664d120c62f182 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:30.905411 3876 tablet_bootstrap.cc:492] T 4a3f86de558243eea0664d120c62f182 P d7449bb1df214bf4a5f93e4631312f08: Bootstrap complete.
I20240502 01:55:30.905793 3876 ts_tablet_manager.cc:1367] T 4a3f86de558243eea0664d120c62f182 P d7449bb1df214bf4a5f93e4631312f08: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.009s
I20240502 01:55:30.906879 3876 raft_consensus.cc:348] T 4a3f86de558243eea0664d120c62f182 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:30.909155 3876 raft_consensus.cc:729] T 4a3f86de558243eea0664d120c62f182 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7449bb1df214bf4a5f93e4631312f08, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.909576 3876 consensus_queue.cc:260] T 4a3f86de558243eea0664d120c62f182 P d7449bb1df214bf4a5f93e4631312f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:30.912994 3876 ts_tablet_manager.cc:1398] T 4a3f86de558243eea0664d120c62f182 P d7449bb1df214bf4a5f93e4631312f08: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 01:55:30.915001 22140 tablet_copy_service.cc:342] P c64840657cdf4b93980bd34120c94d9a: Request end of tablet copy session d7449bb1df214bf4a5f93e4631312f08-4a3f86de558243eea0664d120c62f182 received from {username='slave'} at 127.14.212.199:47983
I20240502 01:55:30.915148 22140 tablet_copy_service.cc:433] P c64840657cdf4b93980bd34120c94d9a: ending tablet copy session d7449bb1df214bf4a5f93e4631312f08-4a3f86de558243eea0664d120c62f182 on tablet 4a3f86de558243eea0664d120c62f182 with peer d7449bb1df214bf4a5f93e4631312f08
W20240502 01:55:30.932992 24802 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.198:42375, user_credentials={real_user=slave}} blocked reactor thread for 51914.4us
I20240502 01:55:30.937404 3897 tablet_copy_client.cc:308] T 88457f21ad4a4e00bb372479c13963ec P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.195:38879
I20240502 01:55:30.938207 22140 tablet_copy_service.cc:140] P c64840657cdf4b93980bd34120c94d9a: Received BeginTabletCopySession request for tablet 88457f21ad4a4e00bb372479c13963ec from peer 0c64d40b5d7644bd8b29feaa80eab72f ({username='slave'} at 127.14.212.197:58377)
I20240502 01:55:30.938401 22140 tablet_copy_service.cc:161] P c64840657cdf4b93980bd34120c94d9a: Beginning new tablet copy session on tablet 88457f21ad4a4e00bb372479c13963ec from peer 0c64d40b5d7644bd8b29feaa80eab72f at {username='slave'} at 127.14.212.197:58377: session id = 0c64d40b5d7644bd8b29feaa80eab72f-88457f21ad4a4e00bb372479c13963ec
I20240502 01:55:30.941430 22140 tablet_copy_source_session.cc:205] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:30.942968 3897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88457f21ad4a4e00bb372479c13963ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.956538 3897 tablet_copy_client.cc:783] T 88457f21ad4a4e00bb372479c13963ec P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:30.960687 3890 tablet_bootstrap.cc:492] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:30.962224 3890 tablet_bootstrap.cc:492] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b: Bootstrap complete.
I20240502 01:55:30.962607 3890 ts_tablet_manager.cc:1367] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent bootstrapping tablet: real 0.085s user 0.004s sys 0.004s
I20240502 01:55:30.964154 3890 raft_consensus.cc:348] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:30.964486 3890 raft_consensus.cc:729] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.965286 3897 tablet_copy_client.cc:647] T 88457f21ad4a4e00bb372479c13963ec P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:30.967561 3897 tablet_copy_client.cc:523] T 88457f21ad4a4e00bb372479c13963ec P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:30.964812 3890 consensus_queue.cc:260] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:30.979224 3890 ts_tablet_manager.cc:1398] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 01:55:30.981568 24888 tablet_copy_service.cc:342] P d7449bb1df214bf4a5f93e4631312f08: Request end of tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-458b19e5186f4ce79f618a10e101c605 received from {username='slave'} at 127.14.212.200:33739
I20240502 01:55:30.981729 24888 tablet_copy_service.cc:433] P d7449bb1df214bf4a5f93e4631312f08: ending tablet copy session 41c70f373c9d49a49891b03a12c6cb2b-458b19e5186f4ce79f618a10e101c605 on tablet 458b19e5186f4ce79f618a10e101c605 with peer 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:55:30.984777 3897 tablet_bootstrap.cc:492] T 88457f21ad4a4e00bb372479c13963ec P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:55:31.008668 3885 tablet_copy_client.cc:783] T ba18c000cd3f42978ed8fd01a4ded60c P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:31.013132 3885 tablet_copy_client.cc:647] T ba18c000cd3f42978ed8fd01a4ded60c P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:31.019393 3885 tablet_copy_client.cc:523] T ba18c000cd3f42978ed8fd01a4ded60c P 759257ea7bef415ca698e60d82199d77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:31.025825 3885 tablet_bootstrap.cc:492] T ba18c000cd3f42978ed8fd01a4ded60c P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:55:31.038739 24867 raft_consensus.cc:1203] T 4a3f86de558243eea0664d120c62f182 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:31.056589 3897 tablet_bootstrap.cc:492] T 88457f21ad4a4e00bb372479c13963ec P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:31.057144 3897 tablet_bootstrap.cc:492] T 88457f21ad4a4e00bb372479c13963ec P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap complete.
I20240502 01:55:31.057452 3897 ts_tablet_manager.cc:1367] T 88457f21ad4a4e00bb372479c13963ec P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.073s user 0.046s sys 0.012s
I20240502 01:55:31.057993 3885 tablet_bootstrap.cc:492] T ba18c000cd3f42978ed8fd01a4ded60c P 759257ea7bef415ca698e60d82199d77: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:31.058262 3897 raft_consensus.cc:348] T 88457f21ad4a4e00bb372479c13963ec P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:31.058522 3897 raft_consensus.cc:729] T 88457f21ad4a4e00bb372479c13963ec P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:55:31.058682 3885 tablet_bootstrap.cc:492] T ba18c000cd3f42978ed8fd01a4ded60c P 759257ea7bef415ca698e60d82199d77: Bootstrap complete.
I20240502 01:55:31.058799 3897 consensus_queue.cc:260] T 88457f21ad4a4e00bb372479c13963ec P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:31.059017 3885 ts_tablet_manager.cc:1367] T ba18c000cd3f42978ed8fd01a4ded60c P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.004s
I20240502 01:55:31.059480 3897 ts_tablet_manager.cc:1398] T 88457f21ad4a4e00bb372479c13963ec P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:55:31.060213 22140 tablet_copy_service.cc:342] P c64840657cdf4b93980bd34120c94d9a: Request end of tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-88457f21ad4a4e00bb372479c13963ec received from {username='slave'} at 127.14.212.197:58377
I20240502 01:55:31.060357 22140 tablet_copy_service.cc:433] P c64840657cdf4b93980bd34120c94d9a: ending tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-88457f21ad4a4e00bb372479c13963ec on tablet 88457f21ad4a4e00bb372479c13963ec with peer 0c64d40b5d7644bd8b29feaa80eab72f
I20240502 01:55:31.062508 3885 raft_consensus.cc:348] T ba18c000cd3f42978ed8fd01a4ded60c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:31.062716 3885 raft_consensus.cc:729] T ba18c000cd3f42978ed8fd01a4ded60c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:55:31.063019 3885 consensus_queue.cc:260] T ba18c000cd3f42978ed8fd01a4ded60c P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:31.063820 3885 ts_tablet_manager.cc:1398] T ba18c000cd3f42978ed8fd01a4ded60c P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:55:31.065369 22271 tablet_copy_service.cc:342] P c2a28baa7ee2469294d94b7fe5455cff: Request end of tablet copy session 759257ea7bef415ca698e60d82199d77-ba18c000cd3f42978ed8fd01a4ded60c received from {username='slave'} at 127.14.212.193:44431
I20240502 01:55:31.065501 22271 tablet_copy_service.cc:433] P c2a28baa7ee2469294d94b7fe5455cff: ending tablet copy session 759257ea7bef415ca698e60d82199d77-ba18c000cd3f42978ed8fd01a4ded60c on tablet ba18c000cd3f42978ed8fd01a4ded60c with peer 759257ea7bef415ca698e60d82199d77
I20240502 01:55:31.124163 25001 raft_consensus.cc:1203] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:31.346227 21857 raft_consensus.cc:1203] T ba18c000cd3f42978ed8fd01a4ded60c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:31.360461 24599 raft_consensus.cc:1203] T 88457f21ad4a4e00bb372479c13963ec P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:55:31.424546 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:31.460012 24583 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:31.472841 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:31.508054 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:31.552111 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:31.597084 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 01:55:31.606770 24668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.194:44411, user_credentials={real_user=slave}} blocked reactor thread for 80422.6us
I20240502 01:55:31.695209 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:31.739398 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:31.892488 24867 consensus_queue.cc:237] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:31.904609 22249 raft_consensus.cc:1261] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:31.904657 25001 raft_consensus.cc:1261] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:31.905360 3492 consensus_queue.cc:1035] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:31.905943 3492 consensus_queue.cc:1035] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:31.915526 3715 raft_consensus.cc:2932] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:31.924479 25001 raft_consensus.cc:2932] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:31.929018 21712 catalog_manager.cc:5484] T 458b19e5186f4ce79f618a10e101c605 P d7449bb1df214bf4a5f93e4631312f08 reported cstate change: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:31.950783 22249 raft_consensus.cc:2932] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:32.002027 24715 tablet_service.cc:1514] Processing DeleteTablet for tablet 458b19e5186f4ce79f618a10e101c605 with delete_type TABLET_DATA_TOMBSTONED (TS 28157bc3e678428fb5f93c077d528513 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41650
I20240502 01:55:32.005443 4005 tablet_replica.cc:330] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:55:32.005757 4005 raft_consensus.cc:2229] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:32.006142 4005 raft_consensus.cc:2258] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:32.008060 4005 ts_tablet_manager.cc:1875] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:32.015432 4005 ts_tablet_manager.cc:1888] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:32.015573 4005 log.cc:1196] T 458b19e5186f4ce79f618a10e101c605 P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/458b19e5186f4ce79f618a10e101c605
I20240502 01:55:32.016386 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet 458b19e5186f4ce79f618a10e101c605 (table rebalancer_extra_table_40 [id=7771fca28431450880af2d1b69228f23]) successfully deleted
I20240502 01:55:32.362984 24583 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:32.382086 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:32.444114 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:32.445806 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:32.458076 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:32.461580 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:32.473592 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:32.533721 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:32.690819 22249 consensus_queue.cc:237] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:32.693346 24599 raft_consensus.cc:1261] T ba18c000cd3f42978ed8fd01a4ded60c P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:32.693379 21857 raft_consensus.cc:1261] T ba18c000cd3f42978ed8fd01a4ded60c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:32.693907 3711 consensus_queue.cc:1035] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:32.694185 3781 consensus_queue.cc:1035] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:32.698827 3711 raft_consensus.cc:2932] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:32.699512 24599 raft_consensus.cc:2932] T ba18c000cd3f42978ed8fd01a4ded60c P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:32.704638 21857 raft_consensus.cc:2932] T ba18c000cd3f42978ed8fd01a4ded60c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:32.705763 21713 catalog_manager.cc:5484] T ba18c000cd3f42978ed8fd01a4ded60c P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: config changed from index 2 to 3, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:32.736052 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet ba18c000cd3f42978ed8fd01a4ded60c with delete_type TABLET_DATA_TOMBSTONED (TS c64840657cdf4b93980bd34120c94d9a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40366
I20240502 01:55:32.741856 4097 tablet_replica.cc:330] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:55:32.742327 4097 raft_consensus.cc:2229] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:32.742551 4097 raft_consensus.cc:2258] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:32.743216 4097 ts_tablet_manager.cc:1875] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:32.748557 4097 ts_tablet_manager.cc:1888] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:32.748668 4097 log.cc:1196] T ba18c000cd3f42978ed8fd01a4ded60c P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/ba18c000cd3f42978ed8fd01a4ded60c
I20240502 01:55:32.749459 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet ba18c000cd3f42978ed8fd01a4ded60c (table rebalancer_extra_table_50 [id=41d42a6d1aa141c98532c43f5e0b1289]) successfully deleted
W20240502 01:55:33.005450 24668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.193:33785, user_credentials={real_user=slave}} blocked reactor thread for 88956.9us
W20240502 01:55:33.028236 21924 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.196:43093, user_credentials={real_user=slave}} blocked reactor thread for 97830.8us
I20240502 01:55:33.141889 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:33.153050 24583 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:33.257445 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:33.257620 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:33.338974 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:33.370951 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:33.405282 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:33.406863 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:33.600986 22118 consensus_queue.cc:237] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:33.603056 24867 raft_consensus.cc:1261] T 4a3f86de558243eea0664d120c62f182 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:33.603375 22249 raft_consensus.cc:1261] T 4a3f86de558243eea0664d120c62f182 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:33.603701 3764 consensus_queue.cc:1035] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:33.603992 3999 consensus_queue.cc:1035] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:33.609268 3764 raft_consensus.cc:2932] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:33.616005 22249 raft_consensus.cc:2932] T 4a3f86de558243eea0664d120c62f182 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:33.617449 24867 raft_consensus.cc:2932] T 4a3f86de558243eea0664d120c62f182 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:33.624867 21707 catalog_manager.cc:5484] T 4a3f86de558243eea0664d120c62f182 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index 2 to 3, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) evicted. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:33.657652 21969 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a3f86de558243eea0664d120c62f182 with delete_type TABLET_DATA_TOMBSTONED (TS 06543b05a16f42099bc86db2a0284412 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46720
I20240502 01:55:33.658440 4190 tablet_replica.cc:330] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412: stopping tablet replica
I20240502 01:55:33.658861 4190 raft_consensus.cc:2229] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:33.659260 4190 raft_consensus.cc:2258] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:33.660996 4190 ts_tablet_manager.cc:1875] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:33.670413 4190 ts_tablet_manager.cc:1888] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:33.670670 4190 log.cc:1196] T 4a3f86de558243eea0664d120c62f182 P 06543b05a16f42099bc86db2a0284412: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/wals/4a3f86de558243eea0664d120c62f182
I20240502 01:55:33.671532 21699 catalog_manager.cc:4830] TS 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): tablet 4a3f86de558243eea0664d120c62f182 (table rebalancer_extra_table_38 [id=b6620e4ad5274690b79a7b8e7e0523bb]) successfully deleted
I20240502 01:55:33.927054 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:33.958812 24583 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:34.005947 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:34.045181 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:34.053193 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:34.058377 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:34.080487 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:34.131487 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:34.260417 21988 consensus_queue.cc:237] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:34.262899 25000 raft_consensus.cc:1261] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:34.262879 24868 raft_consensus.cc:1261] T 37827d24c61a465f88b679cd694df62f P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:34.263485 4138 consensus_queue.cc:1035] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:34.263825 4136 consensus_queue.cc:1035] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:34.266680 4138 raft_consensus.cc:2932] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:34.267864 25000 raft_consensus.cc:2932] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:34.270972 24868 raft_consensus.cc:2932] T 37827d24c61a465f88b679cd694df62f P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:34.276418 21711 catalog_manager.cc:5484] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: config changed from index 2 to 3, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) evicted. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:34.286067 24715 tablet_service.cc:1514] Processing DeleteTablet for tablet 37827d24c61a465f88b679cd694df62f with delete_type TABLET_DATA_TOMBSTONED (TS 28157bc3e678428fb5f93c077d528513 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41650
I20240502 01:55:34.286790 4270 tablet_replica.cc:330] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513: stopping tablet replica
I20240502 01:55:34.287163 4270 raft_consensus.cc:2229] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:34.287408 4270 raft_consensus.cc:2258] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:34.288097 4270 ts_tablet_manager.cc:1875] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:34.293700 4270 ts_tablet_manager.cc:1888] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:34.293841 4270 log.cc:1196] T 37827d24c61a465f88b679cd694df62f P 28157bc3e678428fb5f93c077d528513: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/wals/37827d24c61a465f88b679cd694df62f
I20240502 01:55:34.294675 21700 catalog_manager.cc:4830] TS 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): tablet 37827d24c61a465f88b679cd694df62f (table rebalancer_extra_table_48 [id=ca142102a7e44673b2cac39cee9b5bcd]) successfully deleted
I20240502 01:55:34.643538 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling FlushMRSOp(dcc8171c991b48ddbfda8007e3a87a38): perf score=0.033343
I20240502 01:55:34.650787 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:34.653529 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:34.657155 24583 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:34.686844 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:34.698861 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:34.728024 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:34.756191 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling FlushMRSOp(0eed9ee06a494588a32b4c951d6d3a14): perf score=0.033366
I20240502 01:55:34.807847 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling FlushMRSOp(dcc8171c991b48ddbfda8007e3a87a38): perf score=0.033388
I20240502 01:55:34.833169 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:34.868088 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling FlushMRSOp(3cf315f0117444ae9c5390a1e19460d9): perf score=0.033397
I20240502 01:55:34.915253 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:35.173141 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: FlushMRSOp(dcc8171c991b48ddbfda8007e3a87a38) complete. Timing: real 0.364s user 0.119s sys 0.041s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":14930,"dirs.queue_time_us":12967,"dirs.run_cpu_time_us":256,"dirs.run_wall_time_us":20998,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":20575,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":105728,"threads_started":3,"wal-append.queue_time_us":77059}
I20240502 01:55:35.195192 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling UndoDeltaBlockGCOp(dcc8171c991b48ddbfda8007e3a87a38): 4918 bytes on disk
I20240502 01:55:35.212561 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: UndoDeltaBlockGCOp(dcc8171c991b48ddbfda8007e3a87a38) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240502 01:55:35.290159 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: FlushMRSOp(0eed9ee06a494588a32b4c951d6d3a14) complete. Timing: real 0.533s user 0.084s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":16432,"dirs.queue_time_us":5037,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":2060,"drs_written":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1050,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":11648,"thread_start_us":54695,"threads_started":3,"wal-append.queue_time_us":6332}
I20240502 01:55:35.293599 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling FlushMRSOp(3cf315f0117444ae9c5390a1e19460d9): perf score=0.033510
I20240502 01:55:35.297720 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: FlushMRSOp(dcc8171c991b48ddbfda8007e3a87a38) complete. Timing: real 0.652s user 0.070s sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":26684,"dirs.queue_time_us":20401,"dirs.run_cpu_time_us":270,"dirs.run_wall_time_us":1601,"drs_written":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1120,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":71306,"threads_started":3,"wal-append.queue_time_us":15616}
I20240502 01:55:35.320116 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling UndoDeltaBlockGCOp(dcc8171c991b48ddbfda8007e3a87a38): 4918 bytes on disk
I20240502 01:55:35.324453 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: UndoDeltaBlockGCOp(dcc8171c991b48ddbfda8007e3a87a38) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240502 01:55:35.491991 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: FlushMRSOp(3cf315f0117444ae9c5390a1e19460d9) complete. Timing: real 0.623s user 0.061s sys 0.036s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":31547,"dirs.queue_time_us":64217,"dirs.run_cpu_time_us":324,"dirs.run_wall_time_us":1095,"drs_written":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1011,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":115802,"threads_started":3,"wal-append.queue_time_us":2581}
I20240502 01:55:35.509577 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling UndoDeltaBlockGCOp(3cf315f0117444ae9c5390a1e19460d9): 4834 bytes on disk
I20240502 01:55:35.510754 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: UndoDeltaBlockGCOp(3cf315f0117444ae9c5390a1e19460d9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
I20240502 01:55:35.601302 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: FlushMRSOp(3cf315f0117444ae9c5390a1e19460d9) complete. Timing: real 0.298s user 0.052s sys 0.021s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":14409,"dirs.run_cpu_time_us":306,"dirs.run_wall_time_us":1620,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":15762,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":43814,"threads_started":1,"wal-append.queue_time_us":37352}
I20240502 01:55:35.604331 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling UndoDeltaBlockGCOp(3cf315f0117444ae9c5390a1e19460d9): 4834 bytes on disk
I20240502 01:55:35.605595 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: UndoDeltaBlockGCOp(3cf315f0117444ae9c5390a1e19460d9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":156,"lbm_reads_lt_1ms":4}
I20240502 01:55:35.611191 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling UndoDeltaBlockGCOp(0eed9ee06a494588a32b4c951d6d3a14): 4581 bytes on disk
I20240502 01:55:35.612025 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: UndoDeltaBlockGCOp(0eed9ee06a494588a32b4c951d6d3a14) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240502 01:55:35.674794 24599 consensus_queue.cc:237] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:35.716050 21857 raft_consensus.cc:1261] T 469abc1f70fc456b998461bd367dfc84 P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:55:35.717466 4135 consensus_queue.cc:1035] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:35.764961 22251 raft_consensus.cc:1261] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:55:35.765686 4135 consensus_queue.cc:1035] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:35.781312 22251 raft_consensus.cc:2932] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:35.786478 4347 raft_consensus.cc:2932] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:35.853472 21708 catalog_manager.cc:5484] T 469abc1f70fc456b998461bd367dfc84 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index 2 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New cstate: current_term: 2 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
W20240502 01:55:35.889921 24538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.196:43093, user_credentials={real_user=slave}} blocked reactor thread for 100412us
I20240502 01:55:35.915531 21856 raft_consensus.cc:2932] T 469abc1f70fc456b998461bd367dfc84 P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:35.936841 24981 tablet_service.cc:1514] Processing DeleteTablet for tablet 469abc1f70fc456b998461bd367dfc84 with delete_type TABLET_DATA_TOMBSTONED (TS 41c70f373c9d49a49891b03a12c6cb2b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39964
I20240502 01:55:35.949681 4385 tablet_replica.cc:330] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:55:35.954350 4385 raft_consensus.cc:2229] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:35.954710 4385 raft_consensus.cc:2258] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:35.956049 4385 ts_tablet_manager.cc:1875] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:35.979815 4385 ts_tablet_manager.cc:1888] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 01:55:35.980792 4385 log.cc:1196] T 469abc1f70fc456b998461bd367dfc84 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/469abc1f70fc456b998461bd367dfc84
I20240502 01:55:35.982033 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet 469abc1f70fc456b998461bd367dfc84 (table rebalancer_extra_table_38 [id=b6620e4ad5274690b79a7b8e7e0523bb]) successfully deleted
I20240502 01:55:36.790254 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling FlushMRSOp(db96ccbe81564bb589b0de65977b3321): perf score=0.033341
I20240502 01:55:36.844026 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling FlushMRSOp(da66e78c2c5044b2924682be0b643b0a): perf score=0.033354
I20240502 01:55:36.899088 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling FlushMRSOp(88457f21ad4a4e00bb372479c13963ec): perf score=0.033363
I20240502 01:55:36.982496 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: FlushMRSOp(db96ccbe81564bb589b0de65977b3321) complete. Timing: real 0.192s user 0.049s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":11283,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":15779,"drs_written":1,"lbm_read_time_us":155,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1041,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":42419,"threads_started":2,"wal-append.queue_time_us":30851}
I20240502 01:55:36.983044 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling FlushMRSOp(88457f21ad4a4e00bb372479c13963ec): perf score=0.033395
I20240502 01:55:36.985697 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling UndoDeltaBlockGCOp(db96ccbe81564bb589b0de65977b3321): 4922 bytes on disk
I20240502 01:55:37.011384 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: UndoDeltaBlockGCOp(db96ccbe81564bb589b0de65977b3321) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4}
I20240502 01:55:37.015739 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling FlushMRSOp(da66e78c2c5044b2924682be0b643b0a): perf score=0.033399
I20240502 01:55:37.065119 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: FlushMRSOp(da66e78c2c5044b2924682be0b643b0a) complete. Timing: real 0.221s user 0.061s sys 0.030s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":125649,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":2857,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1105,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":141897,"threads_started":2,"wal-append.queue_time_us":8728}
I20240502 01:55:37.067858 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling FlushMRSOp(88457f21ad4a4e00bb372479c13963ec): perf score=0.033410
I20240502 01:55:37.102608 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:37.189210 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: FlushMRSOp(88457f21ad4a4e00bb372479c13963ec) complete. Timing: real 0.289s user 0.050s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":30576,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1391,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2399,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":40443,"threads_started":2,"wal-append.queue_time_us":13506}
I20240502 01:55:37.191505 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling UndoDeltaBlockGCOp(88457f21ad4a4e00bb372479c13963ec): 4835 bytes on disk
I20240502 01:55:37.193305 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: UndoDeltaBlockGCOp(88457f21ad4a4e00bb372479c13963ec) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":174,"lbm_reads_lt_1ms":4}
I20240502 01:55:37.261484 22229 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:37.275988 24847 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:55:37.284823 22100 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:55:37.285459 21837 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:37.326478 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:37.369004 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: FlushMRSOp(88457f21ad4a4e00bb372479c13963ec) complete. Timing: real 0.301s user 0.056s sys 0.038s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":752,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":3574,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1006,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":190774,"threads_started":1,"wal-append.queue_time_us":187736}
I20240502 01:55:37.372030 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling UndoDeltaBlockGCOp(88457f21ad4a4e00bb372479c13963ec): 4835 bytes on disk
I20240502 01:55:37.373834 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: UndoDeltaBlockGCOp(88457f21ad4a4e00bb372479c13963ec) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":483,"lbm_reads_lt_1ms":4}
I20240502 01:55:37.377432 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: FlushMRSOp(da66e78c2c5044b2924682be0b643b0a) complete. Timing: real 0.361s user 0.051s sys 0.037s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":191,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1781,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3908,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":2816,"thread_start_us":163611,"threads_started":1,"wal-append.queue_time_us":163756}
I20240502 01:55:37.386612 24715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:55:37.386212 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling UndoDeltaBlockGCOp(da66e78c2c5044b2924682be0b643b0a): 4582 bytes on disk
I20240502 01:55:37.388239 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: UndoDeltaBlockGCOp(da66e78c2c5044b2924682be0b643b0a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
I20240502 01:55:37.396766 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling UndoDeltaBlockGCOp(da66e78c2c5044b2924682be0b643b0a): 4582 bytes on disk
I20240502 01:55:37.415954 24583 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:37.450925 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: FlushMRSOp(88457f21ad4a4e00bb372479c13963ec) complete. Timing: real 0.468s user 0.034s sys 0.022s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":11652,"dirs.run_cpu_time_us":268,"dirs.run_wall_time_us":23271,"drs_written":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1142,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":17939,"threads_started":2,"wal-append.queue_time_us":8532}
I20240502 01:55:37.453583 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling UndoDeltaBlockGCOp(88457f21ad4a4e00bb372479c13963ec): 4835 bytes on disk
I20240502 01:55:37.454316 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: UndoDeltaBlockGCOp(88457f21ad4a4e00bb372479c13963ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240502 01:55:37.456647 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling FlushMRSOp(db96ccbe81564bb589b0de65977b3321): perf score=0.033524
I20240502 01:55:37.478040 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: UndoDeltaBlockGCOp(da66e78c2c5044b2924682be0b643b0a) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4}
I20240502 01:55:37.569360 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: FlushMRSOp(db96ccbe81564bb589b0de65977b3321) complete. Timing: real 0.111s user 0.052s sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1344,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":998,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":21959,"threads_started":1,"wal-append.queue_time_us":22046}
I20240502 01:55:37.571906 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling UndoDeltaBlockGCOp(db96ccbe81564bb589b0de65977b3321): 4922 bytes on disk
I20240502 01:55:37.572634 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: UndoDeltaBlockGCOp(db96ccbe81564bb589b0de65977b3321) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240502 01:55:37.664065 21985 consensus_queue.cc:237] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:37.666625 22251 raft_consensus.cc:1261] T 992544bbea6547de96298fd32d74b232 P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:55:37.666957 21857 raft_consensus.cc:1261] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:55:37.667119 4136 consensus_queue.cc:1035] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:37.667654 4136 consensus_queue.cc:1035] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:37.670181 4501 raft_consensus.cc:2932] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:37.670784 22251 raft_consensus.cc:2932] T 992544bbea6547de96298fd32d74b232 P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:37.677493 21708 catalog_manager.cc:5484] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index 4 to 6, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New cstate: current_term: 2 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:37.680436 21857 raft_consensus.cc:2932] T 992544bbea6547de96298fd32d74b232 P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:37.700629 24981 tablet_service.cc:1514] Processing DeleteTablet for tablet 992544bbea6547de96298fd32d74b232 with delete_type TABLET_DATA_TOMBSTONED (TS 41c70f373c9d49a49891b03a12c6cb2b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39964
I20240502 01:55:37.702582 4533 tablet_replica.cc:330] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:55:37.702894 4533 raft_consensus.cc:2229] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:37.703244 4533 raft_consensus.cc:2258] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:37.704012 4533 ts_tablet_manager.cc:1875] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:37.711591 4533 ts_tablet_manager.cc:1888] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 01:55:37.711819 4533 log.cc:1196] T 992544bbea6547de96298fd32d74b232 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/992544bbea6547de96298fd32d74b232
I20240502 01:55:37.712867 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet 992544bbea6547de96298fd32d74b232 (table rebalancer_extra_table_4 [id=c1a2048ad81b49729677603cbe361089]) successfully deleted
I20240502 01:55:37.788671 21857 consensus_queue.cc:237] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:37.791805 21985 raft_consensus.cc:1261] T 5b3f40a9f1c34644b787905c28a54b29 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:37.791904 21791 consensus_peers.cc:485] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b3f40a9f1c34644b787905c28a54b29. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:37.792460 3952 consensus_queue.cc:1035] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:37.793016 22120 raft_consensus.cc:1261] T 5b3f40a9f1c34644b787905c28a54b29 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:37.794579 3952 consensus_queue.cc:1035] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:37.809754 3952 raft_consensus.cc:2932] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:37.810601 22120 raft_consensus.cc:2932] T 5b3f40a9f1c34644b787905c28a54b29 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:37.813885 21985 raft_consensus.cc:2932] T 5b3f40a9f1c34644b787905c28a54b29 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:37.816973 21707 catalog_manager.cc:5484] T 5b3f40a9f1c34644b787905c28a54b29 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index -1 to 4, VOTER d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) added. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:37.857501 25000 consensus_queue.cc:237] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
W20240502 01:55:37.861366 24933 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: feffa925599b4d6aafa48e09e4f6467a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:37.862403 24867 raft_consensus.cc:1261] T feffa925599b4d6aafa48e09e4f6467a P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:37.863219 24603 raft_consensus.cc:1261] T feffa925599b4d6aafa48e09e4f6467a P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:37.863162 4246 consensus_queue.cc:1035] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:37.871886 4246 consensus_queue.cc:1035] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:37.887820 4246 raft_consensus.cc:2932] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:37.888451 24603 raft_consensus.cc:2932] T feffa925599b4d6aafa48e09e4f6467a P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:37.895565 21708 catalog_manager.cc:5484] T feffa925599b4d6aafa48e09e4f6467a P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:37.908443 24867 raft_consensus.cc:2932] T feffa925599b4d6aafa48e09e4f6467a P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:37.985626 24999 consensus_queue.cc:237] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:37.988411 21857 raft_consensus.cc:1261] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:37.988406 21988 raft_consensus.cc:1261] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:37.989086 4590 consensus_queue.cc:1035] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:37.989496 4246 consensus_queue.cc:1035] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:55:37.989749 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f018db81d6214bd0ac12feaf54541fed. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:37.992764 4590 raft_consensus.cc:2932] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:37.993507 21988 raft_consensus.cc:2932] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:37.997685 21857 raft_consensus.cc:2932] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:38.007243 21708 catalog_manager.cc:5484] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:38.058735 24867 consensus_queue.cc:237] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:38.061192 22251 raft_consensus.cc:1261] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:55:38.061376 24802 consensus_peers.cc:485] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 -> Peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Couldn't send request to peer 0c64d40b5d7644bd8b29feaa80eab72f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36d358202b544e74bcc378060082df4d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:38.061371 21857 raft_consensus.cc:1261] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer d7449bb1df214bf4a5f93e4631312f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:38.061836 4341 consensus_queue.cc:1035] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:38.062170 4402 consensus_queue.cc:1035] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:38.074378 4402 raft_consensus.cc:2932] T 36d358202b544e74bcc378060082df4d P d7449bb1df214bf4a5f93e4631312f08 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:38.075111 21857 raft_consensus.cc:2932] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:38.082270 22251 raft_consensus.cc:2932] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:38.083328 21712 catalog_manager.cc:5484] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index -1 to 2, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New cstate: current_term: 1 leader_uuid: "d7449bb1df214bf4a5f93e4631312f08" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:38.123095 22251 consensus_queue.cc:237] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:38.146642 25000 raft_consensus.cc:1261] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:38.149097 21988 raft_consensus.cc:1261] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:38.150116 4470 consensus_queue.cc:1035] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:38.150723 4470 consensus_queue.cc:1035] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:55:38.154798 22186 consensus_peers.cc:485] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad8bc5be633d4646a7c10d2483ed4e5b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:38.162343 4647 raft_consensus.cc:2932] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:38.165119 25000 raft_consensus.cc:2932] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:38.168212 21988 raft_consensus.cc:2932] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:38.172835 21707 catalog_manager.cc:5484] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:38.218964 24601 consensus_queue.cc:237] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:38.221817 24867 raft_consensus.cc:1261] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:38.222400 4667 consensus_queue.cc:1035] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:55:38.222708 24536 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b0e8db8c3414ce0a5ff12200fe6b544. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:38.222783 24733 raft_consensus.cc:1261] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:38.223305 4667 consensus_queue.cc:1035] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:38.226471 4347 raft_consensus.cc:2932] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:38.227077 24733 raft_consensus.cc:2932] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:38.231784 24867 raft_consensus.cc:2932] T 2b0e8db8c3414ce0a5ff12200fe6b544 P d7449bb1df214bf4a5f93e4631312f08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:38.233253 21708 catalog_manager.cc:5484] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 28157bc3e678428fb5f93c077d528513 reported cstate change: config changed from index -1 to 2, VOTER 06543b05a16f42099bc86db2a0284412 (127.14.212.194) added. New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:38.288796 24603 consensus_queue.cc:237] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:38.291730 22118 raft_consensus.cc:1261] T 2036e577d94242728ae4d00fe845121b P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:55:38.292052 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2036e577d94242728ae4d00fe845121b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:38.293077 21854 raft_consensus.cc:1261] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:38.293231 4500 consensus_queue.cc:1035] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:38.293903 4500 consensus_queue.cc:1035] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:38.296857 4667 raft_consensus.cc:2932] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:38.297521 22120 raft_consensus.cc:2932] T 2036e577d94242728ae4d00fe845121b P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:38.301026 4688 ts_tablet_manager.cc:902] T feffa925599b4d6aafa48e09e4f6467a P 06543b05a16f42099bc86db2a0284412: Initiating tablet copy from peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:55:38.302078 21854 raft_consensus.cc:2932] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:38.309195 21707 catalog_manager.cc:5484] T 2036e577d94242728ae4d00fe845121b P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:38.376334 4688 tablet_copy_client.cc:308] T feffa925599b4d6aafa48e09e4f6467a P 06543b05a16f42099bc86db2a0284412: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.200:37097
I20240502 01:55:38.377689 25021 tablet_copy_service.cc:140] P 41c70f373c9d49a49891b03a12c6cb2b: Received BeginTabletCopySession request for tablet feffa925599b4d6aafa48e09e4f6467a from peer 06543b05a16f42099bc86db2a0284412 ({username='slave'} at 127.14.212.194:34591)
I20240502 01:55:38.377911 25021 tablet_copy_service.cc:161] P 41c70f373c9d49a49891b03a12c6cb2b: Beginning new tablet copy session on tablet feffa925599b4d6aafa48e09e4f6467a from peer 06543b05a16f42099bc86db2a0284412 at {username='slave'} at 127.14.212.194:34591: session id = 06543b05a16f42099bc86db2a0284412-feffa925599b4d6aafa48e09e4f6467a
I20240502 01:55:38.385003 25021 tablet_copy_source_session.cc:205] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:38.387508 4688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feffa925599b4d6aafa48e09e4f6467a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:38.402045 4688 tablet_copy_client.cc:783] T feffa925599b4d6aafa48e09e4f6467a P 06543b05a16f42099bc86db2a0284412: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:38.402323 4688 tablet_copy_client.cc:647] T feffa925599b4d6aafa48e09e4f6467a P 06543b05a16f42099bc86db2a0284412: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:38.404461 4688 tablet_copy_client.cc:523] T feffa925599b4d6aafa48e09e4f6467a P 06543b05a16f42099bc86db2a0284412: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:38.408011 4688 tablet_bootstrap.cc:492] T feffa925599b4d6aafa48e09e4f6467a P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:55:38.411993 4704 ts_tablet_manager.cc:902] T 5b3f40a9f1c34644b787905c28a54b29 P d7449bb1df214bf4a5f93e4631312f08: Initiating tablet copy from peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
F20240502 01:55:38.412437 4704 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-071ad932ce95499fac4d7af7292b42ff.queue_time_us, tablet-download-pool-071ad932ce95499fac4d7af7292b42ff.queue_time_us) (tablet-download-pool-071ad932ce95499fac4d7af7292b42ff.run_cpu_time_us, tablet-download-pool-071ad932ce95499fac4d7af7292b42ff.run_cpu_time_us) (tablet-download-pool-071ad932ce95499fac4d7af7292b42ff.run_wall_time_us, tablet-download-pool-071ad932ce95499fac4d7af7292b42ff.run_wall_time_us) (tablet-download-pool-0eed9ee06a494588a32b4c951d6d3a14.queue_time_us, tablet-download-pool-0eed9ee06a494588a32b4c951d6d3a14.queue_time_us) (tablet-download-pool-0eed9ee06a494588a32b4c951d6d3a14.run_cpu_time_us, tablet-download-pool-0eed9ee06a494588a32b4c951d6d3a14.run_cpu_time_us) (tablet-download-pool-0eed9ee06a494588a32b4c951d6d3a14.run_wall_time_us, tablet-download-pool-0eed9ee06a494588a32b4c951d6d3a14.run_wall_time_us) (tablet-download-pool-0f8200b044ef44e1a0451baf2b1485d2.queue_time_us, tablet-download-pool-0f8200b044ef44e1a0451baf2b1485d2.queue_time_us) (tablet-download-pool-0f8200b044ef44e1a0451baf2b1485d2.run_cpu_time_us, tablet-download-pool-0f8200b044ef44e1a0451baf2b1485d2.run_cpu_time_us) (tablet-download-pool-0f8200b044ef44e1a0451baf2b1485d2.run_wall_time_us, tablet-download-pool-0f8200b044ef44e1a0451baf2b1485d2.run_wall_time_us) (tablet-download-pool-37827d24c61a465f88b679cd694df62f.queue_time_us, tablet-download-pool-37827d24c61a465f88b679cd694df62f.queue_time_us) (tablet-download-pool-37827d24c61a465f88b679cd694df62f.run_cpu_time_us, tablet-download-pool-37827d24c61a465f88b679cd694df62f.run_cpu_time_us) (tablet-download-pool-37827d24c61a465f88b679cd694df62f.run_wall_time_us, tablet-download-pool-37827d24c61a465f88b679cd694df62f.run_wall_time_us) (tablet-download-pool-3a27dc6af117475cb227e3526d2a9007.queue_time_us, tablet-download-pool-3a27dc6af117475cb227e3526d2a9007.queue_time_us) (tablet-download-pool-3a27dc6af117475cb227e3526d2a9007.run_cpu_time_us, tablet-download-pool-3a27dc6af117475cb227e3526d2a9007.run_cpu_time_us) (tablet-download-pool-3a27dc6af117475cb227e3526d2a9007.run_wall_time_us, tablet-download-pool-3a27dc6af117475cb227e3526d2a9007.run_wall_time_us) (tablet-download-pool-4a3f86de558243eea0664d120c62f182.queue_time_us, tablet-download-pool-4a3f86de558243eea0664d120c62f182.queue_time_us) (tablet-download-pool-4a3f86de558243eea0664d120c62f182.run_cpu_time_us, tablet-download-pool-4a3f86de558243eea0664d120c62f182.run_cpu_time_us) (tablet-download-pool-4a3f86de558243eea0664d120c62f182.run_wall_time_us, tablet-download-pool-4a3f86de558243eea0664d120c62f182.run_wall_time_us) (tablet-download-pool-5b3f40a9f1c34644b787905c28a54b29.queue_time_us, tablet-download-pool-5b3f40a9f1c34644b787905c28a54b29.queue_time_us) (tablet-download-pool-9bceab60366b4ade98f86072d391e7f4.queue_time_us, tablet-download-pool-9bceab60366b4ade98f86072d391e7f4.queue_time_us) (tablet-download-pool-9bceab60366b4ade98f86072d391e7f4.run_cpu_time_us, tablet-download-pool-9bceab60366b4ade98f86072d391e7f4.run_cpu_time_us) (tablet-download-pool-9bceab60366b4ade98f86072d391e7f4.run_wall_time_us, tablet-download-pool-9bceab60366b4ade98f86072d391e7f4.run_wall_time_us) (tablet-download-pool-b1c6c622638444bd854616902b8d91ad.queue_time_us, tablet-download-pool-b1c6c622638444bd854616902b8d91ad.queue_time_us) (tablet-download-pool-b1c6c622638444bd854616902b8d91ad.run_cpu_time_us, tablet-download-pool-b1c6c622638444bd854616902b8d91ad.run_cpu_time_us) (tablet-download-pool-b1c6c622638444bd854616902b8d91ad.run_wall_time_us, tablet-download-pool-b1c6c622638444bd854616902b8d91ad.run_wall_time_us) (tablet-download-pool-ba39439749a9486799373e173110f5c5.queue_time_us, tablet-download-pool-ba39439749a9486799373e173110f5c5.queue_time_us) (tablet-download-pool-ba39439749a9486799373e173110f5c5.run_cpu_time_us, tablet-download-pool-ba39439749a9486799373e173110f5c5.run_cpu_time_us) (tablet-download-pool-ba39439749a9486799373e173110f5c5.run_wall_time_us, tablet-download-pool-ba39439749a9486799373e173110f5c5.run_wall_time_us) (tablet-download-pool-c0877e287cb94abf98723449b9805f80.queue_time_us, tablet-download-pool-c0877e287cb94abf98723449b9805f80.queue_time_us) (tablet-download-pool-c0877e287cb94abf98723449b9805f80.run_cpu_time_us, tablet-download-pool-c0877e287cb94abf98723449b9805f80.run_cpu_time_us) (tablet-download-pool-c0877e287cb94abf98723449b9805f80.run_wall_time_us, tablet-download-pool-c0877e287cb94abf98723449b9805f80.run_wall_time_us) (tablet-download-pool-d4b6dfe0e8da4352b136d6f97d473bad.queue_time_us, tablet-download-pool-d4b6dfe0e8da4352b136d6f97d473bad.queue_time_us) (tablet-download-pool-d4b6dfe0e8da4352b136d6f97d473bad.run_cpu_time_us, tablet-download-pool-d4b6dfe0e8da4352b136d6f97d473bad.run_cpu_time_us) (tablet-download-pool-d4b6dfe0e8da4352b136d6f97d473bad.run_wall_time_us, tablet-download-pool-d4b6dfe0e8da4352b136d6f97d473bad.run_wall_time_us) (tablet-download-pool-d81f9cab671a4258aee1dd4ea4a5ee6b.queue_time_us, tablet-download-pool-d81f9cab671a4258aee1dd4ea4a5ee6b.queue_time_us) (tablet-download-pool-d81f9cab671a4258aee1dd4ea4a5ee6b.run_cpu_time_us, tablet-download-pool-d81f9cab671a4258aee1dd4ea4a5ee6b.run_cpu_time_us) (tablet-download-pool-d81f9cab671a4258aee1dd4ea4a5ee6b.run_wall_time_us, tablet-download-pool-d81f9cab671a4258aee1dd4ea4a5ee6b.run_wall_time_us) (tablet-download-pool-db96ccbe81564bb589b0de65977b3321.queue_time_us, tablet-download-pool-db96ccbe81564bb589b0de65977b3321.queue_time_us) (tablet-download-pool-db96ccbe81564bb589b0de65977b3321.run_cpu_time_us, tablet-download-pool-db96ccbe81564bb589b0de65977b3321.run_cpu_time_us) (tablet-download-pool-db96ccbe81564bb589b0de65977b3321.run_wall_time_us, tablet-download-pool-db96ccbe81564bb589b0de65977b3321.run_wall_time_us) (tablet-download-pool-fafa4ef17fac41b68e3bf4aaa8a4caa8.queue_time_us, tablet-download-pool-fafa4ef17fac41b68e3bf4aaa8a4caa8.queue_time_us) (tablet-download-pool-fafa4ef17fac41b68e3bf4aaa8a4caa8.run_cpu_time_us, tablet-download-pool-fafa4ef17fac41b68e3bf4aaa8a4caa8.run_cpu_time_us) (tablet-download-pool-fafa4ef17fac41b68e3bf4aaa8a4caa8.run_wall_time_us, tablet-download-pool-fafa4ef17fac41b68e3bf4aaa8a4caa8.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 01:55:38.415858 4688 tablet_bootstrap.cc:492] T feffa925599b4d6aafa48e09e4f6467a P 06543b05a16f42099bc86db2a0284412: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:38.416247 4688 tablet_bootstrap.cc:492] T feffa925599b4d6aafa48e09e4f6467a P 06543b05a16f42099bc86db2a0284412: Bootstrap complete.
I20240502 01:55:38.416518 4688 ts_tablet_manager.cc:1367] T feffa925599b4d6aafa48e09e4f6467a P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240502 01:55:38.417320 4688 raft_consensus.cc:348] T feffa925599b4d6aafa48e09e4f6467a P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:38.417506 4688 raft_consensus.cc:729] T feffa925599b4d6aafa48e09e4f6467a P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:55:38.417742 4688 consensus_queue.cc:260] T feffa925599b4d6aafa48e09e4f6467a P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
@ 0x7f6d981addcd google::LogMessage::Fail() at ??:0
@ 0x7f6d981b1b93 google::LogMessage::SendToLog() at ??:0
I20240502 01:55:38.421447 4688 ts_tablet_manager.cc:1398] T feffa925599b4d6aafa48e09e4f6467a P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:55:38.422245 25021 tablet_copy_service.cc:342] P 41c70f373c9d49a49891b03a12c6cb2b: Request end of tablet copy session 06543b05a16f42099bc86db2a0284412-feffa925599b4d6aafa48e09e4f6467a received from {username='slave'} at 127.14.212.194:34591
I20240502 01:55:38.422374 25021 tablet_copy_service.cc:433] P 41c70f373c9d49a49891b03a12c6cb2b: ending tablet copy session 06543b05a16f42099bc86db2a0284412-feffa925599b4d6aafa48e09e4f6467a on tablet feffa925599b4d6aafa48e09e4f6467a with peer 06543b05a16f42099bc86db2a0284412
@ 0x7f6d981ad7cc google::LogMessage::Flush() at ??:0
@ 0x7f6d981aef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6d9aeee742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f6d9aebaf8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f6d9aeb4cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 01:55:38.447005 21854 consensus_queue.cc:237] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
@ 0x7f6daa3078de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 01:55:38.451565 21988 raft_consensus.cc:1261] T 72f46ec6ed7848438cb9ad6e38053c4d P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:38.452014 21791 consensus_peers.cc:485] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 -> Peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Couldn't send request to peer 0c64d40b5d7644bd8b29feaa80eab72f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72f46ec6ed7848438cb9ad6e38053c4d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:38.452127 4566 consensus_queue.cc:1035] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:38.453291 22120 raft_consensus.cc:1261] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:38.453912 4566 consensus_queue.cc:1035] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
@ 0x7f6daa31fdaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 01:55:38.462563 3952 raft_consensus.cc:2932] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:38.464607 22120 raft_consensus.cc:2932] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:38.467908 21988 raft_consensus.cc:2932] T 72f46ec6ed7848438cb9ad6e38053c4d P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
@ 0x7f6daa44a480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 01:55:38.471829 21708 catalog_manager.cc:5484] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
@ 0x7f6daa482a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f6daa465981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f6daa46570e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f6d9aebeb75 kudu::ThreadPool::DispatchThread() at ??:0
I20240502 01:55:38.527649 4723 ts_tablet_manager.cc:902] T f018db81d6214bd0ac12feaf54541fed P 28157bc3e678428fb5f93c077d528513: Initiating tablet copy from peer 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
@ 0x7f6d9aec219c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f6d9aec2019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f6d9ae9d01e kudu::Thread::SuperviseThread() at ??:0
I20240502 01:55:38.554908 4727 ts_tablet_manager.cc:902] T 36d358202b544e74bcc378060082df4d P 0c64d40b5d7644bd8b29feaa80eab72f: Initiating tablet copy from peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
@ 0x7f6d9e2226db start_thread at ??:0
I20240502 01:55:38.560482 4723 tablet_copy_client.cc:308] T f018db81d6214bd0ac12feaf54541fed P 28157bc3e678428fb5f93c077d528513: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.200:37097
I20240502 01:55:38.561919 25021 tablet_copy_service.cc:140] P 41c70f373c9d49a49891b03a12c6cb2b: Received BeginTabletCopySession request for tablet f018db81d6214bd0ac12feaf54541fed from peer 28157bc3e678428fb5f93c077d528513 ({username='slave'} at 127.14.212.198:41233)
I20240502 01:55:38.562114 25021 tablet_copy_service.cc:161] P 41c70f373c9d49a49891b03a12c6cb2b: Beginning new tablet copy session on tablet f018db81d6214bd0ac12feaf54541fed from peer 28157bc3e678428fb5f93c077d528513 at {username='slave'} at 127.14.212.198:41233: session id = 28157bc3e678428fb5f93c077d528513-f018db81d6214bd0ac12feaf54541fed
I20240502 01:55:38.564553 25021 tablet_copy_source_session.cc:205] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:38.566079 4723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f018db81d6214bd0ac12feaf54541fed. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f6d9706071f clone at ??:0
I20240502 01:55:38.577880 4723 tablet_copy_client.cc:783] T f018db81d6214bd0ac12feaf54541fed P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:38.578183 4723 tablet_copy_client.cc:647] T f018db81d6214bd0ac12feaf54541fed P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:38.588582 4723 tablet_copy_client.cc:523] T f018db81d6214bd0ac12feaf54541fed P 28157bc3e678428fb5f93c077d528513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:38.592757 4723 tablet_bootstrap.cc:492] T f018db81d6214bd0ac12feaf54541fed P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:55:38.601230 4739 ts_tablet_manager.cc:902] T ad8bc5be633d4646a7c10d2483ed4e5b P c64840657cdf4b93980bd34120c94d9a: Initiating tablet copy from peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:55:38.602800 4739 tablet_copy_client.cc:308] T ad8bc5be633d4646a7c10d2483ed4e5b P c64840657cdf4b93980bd34120c94d9a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.196:43093
I20240502 01:55:38.603649 22271 tablet_copy_service.cc:140] P c2a28baa7ee2469294d94b7fe5455cff: Received BeginTabletCopySession request for tablet ad8bc5be633d4646a7c10d2483ed4e5b from peer c64840657cdf4b93980bd34120c94d9a ({username='slave'} at 127.14.212.195:40447)
I20240502 01:55:38.603801 22271 tablet_copy_service.cc:161] P c2a28baa7ee2469294d94b7fe5455cff: Beginning new tablet copy session on tablet ad8bc5be633d4646a7c10d2483ed4e5b from peer c64840657cdf4b93980bd34120c94d9a at {username='slave'} at 127.14.212.195:40447: session id = c64840657cdf4b93980bd34120c94d9a-ad8bc5be633d4646a7c10d2483ed4e5b
I20240502 01:55:38.607336 22271 tablet_copy_source_session.cc:205] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:38.609300 4739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad8bc5be633d4646a7c10d2483ed4e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:38.615087 4723 tablet_bootstrap.cc:492] T f018db81d6214bd0ac12feaf54541fed P 28157bc3e678428fb5f93c077d528513: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:38.617156 4723 tablet_bootstrap.cc:492] T f018db81d6214bd0ac12feaf54541fed P 28157bc3e678428fb5f93c077d528513: Bootstrap complete.
I20240502 01:55:38.617523 4723 ts_tablet_manager.cc:1367] T f018db81d6214bd0ac12feaf54541fed P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.006s
I20240502 01:55:38.618758 4727 tablet_copy_client.cc:308] T 36d358202b544e74bcc378060082df4d P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.199:35737
I20240502 01:55:38.623525 4723 raft_consensus.cc:348] T f018db81d6214bd0ac12feaf54541fed P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:38.623999 4723 raft_consensus.cc:729] T f018db81d6214bd0ac12feaf54541fed P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:55:38.624281 4723 consensus_queue.cc:260] T f018db81d6214bd0ac12feaf54541fed P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:38.625065 4723 ts_tablet_manager.cc:1398] T f018db81d6214bd0ac12feaf54541fed P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:55:38.626802 25021 tablet_copy_service.cc:342] P 41c70f373c9d49a49891b03a12c6cb2b: Request end of tablet copy session 28157bc3e678428fb5f93c077d528513-f018db81d6214bd0ac12feaf54541fed received from {username='slave'} at 127.14.212.198:41233
I20240502 01:55:38.626935 25021 tablet_copy_service.cc:433] P 41c70f373c9d49a49891b03a12c6cb2b: ending tablet copy session 28157bc3e678428fb5f93c077d528513-f018db81d6214bd0ac12feaf54541fed on tablet f018db81d6214bd0ac12feaf54541fed with peer 28157bc3e678428fb5f93c077d528513
I20240502 01:55:38.627290 4739 tablet_copy_client.cc:783] T ad8bc5be633d4646a7c10d2483ed4e5b P c64840657cdf4b93980bd34120c94d9a: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:38.627532 4739 tablet_copy_client.cc:647] T ad8bc5be633d4646a7c10d2483ed4e5b P c64840657cdf4b93980bd34120c94d9a: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:38.629671 4739 tablet_copy_client.cc:523] T ad8bc5be633d4646a7c10d2483ed4e5b P c64840657cdf4b93980bd34120c94d9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:38.634290 4739 tablet_bootstrap.cc:492] T ad8bc5be633d4646a7c10d2483ed4e5b P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:55:38.643393 4739 tablet_bootstrap.cc:492] T ad8bc5be633d4646a7c10d2483ed4e5b P c64840657cdf4b93980bd34120c94d9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:38.643745 4739 tablet_bootstrap.cc:492] T ad8bc5be633d4646a7c10d2483ed4e5b P c64840657cdf4b93980bd34120c94d9a: Bootstrap complete.
I20240502 01:55:38.643934 4739 ts_tablet_manager.cc:1367] T ad8bc5be633d4646a7c10d2483ed4e5b P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20240502 01:55:38.644479 4739 raft_consensus.cc:348] T ad8bc5be633d4646a7c10d2483ed4e5b P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:38.645357 4739 raft_consensus.cc:729] T ad8bc5be633d4646a7c10d2483ed4e5b P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:55:38.645635 4739 consensus_queue.cc:260] T ad8bc5be633d4646a7c10d2483ed4e5b P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:38.646313 4739 ts_tablet_manager.cc:1398] T ad8bc5be633d4646a7c10d2483ed4e5b P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 01:55:38.650040 22271 tablet_copy_service.cc:342] P c2a28baa7ee2469294d94b7fe5455cff: Request end of tablet copy session c64840657cdf4b93980bd34120c94d9a-ad8bc5be633d4646a7c10d2483ed4e5b received from {username='slave'} at 127.14.212.195:40447
I20240502 01:55:38.650187 22271 tablet_copy_service.cc:433] P c2a28baa7ee2469294d94b7fe5455cff: ending tablet copy session c64840657cdf4b93980bd34120c94d9a-ad8bc5be633d4646a7c10d2483ed4e5b on tablet ad8bc5be633d4646a7c10d2483ed4e5b with peer c64840657cdf4b93980bd34120c94d9a
I20240502 01:55:38.709131 4723 ts_tablet_manager.cc:902] T 2036e577d94242728ae4d00fe845121b P 28157bc3e678428fb5f93c077d528513: Initiating tablet copy from peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:55:38.712322 22118 consensus_queue.cc:237] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:38.731143 25000 raft_consensus.cc:1261] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:38.731681 4592 consensus_queue.cc:1035] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:55:38.731992 22053 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28dbb8d9ef5f4efdb5ba6e8ba9179aa8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:38.735344 4723 tablet_copy_client.cc:308] T 2036e577d94242728ae4d00fe845121b P 28157bc3e678428fb5f93c077d528513: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.197:36677
I20240502 01:55:38.736135 24623 tablet_copy_service.cc:140] P 0c64d40b5d7644bd8b29feaa80eab72f: Received BeginTabletCopySession request for tablet 2036e577d94242728ae4d00fe845121b from peer 28157bc3e678428fb5f93c077d528513 ({username='slave'} at 127.14.212.198:35347)
I20240502 01:55:38.736280 24623 tablet_copy_service.cc:161] P 0c64d40b5d7644bd8b29feaa80eab72f: Beginning new tablet copy session on tablet 2036e577d94242728ae4d00fe845121b from peer 28157bc3e678428fb5f93c077d528513 at {username='slave'} at 127.14.212.198:35347: session id = 28157bc3e678428fb5f93c077d528513-2036e577d94242728ae4d00fe845121b
I20240502 01:55:38.739711 24623 tablet_copy_source_session.cc:205] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:38.741847 4723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2036e577d94242728ae4d00fe845121b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:38.752866 4723 tablet_copy_client.cc:783] T 2036e577d94242728ae4d00fe845121b P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:38.753149 4723 tablet_copy_client.cc:647] T 2036e577d94242728ae4d00fe845121b P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:38.755493 4723 tablet_copy_client.cc:523] T 2036e577d94242728ae4d00fe845121b P 28157bc3e678428fb5f93c077d528513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:38.758538 4723 tablet_bootstrap.cc:492] T 2036e577d94242728ae4d00fe845121b P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:55:38.763478 4723 tablet_bootstrap.cc:492] T 2036e577d94242728ae4d00fe845121b P 28157bc3e678428fb5f93c077d528513: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:38.763746 4723 tablet_bootstrap.cc:492] T 2036e577d94242728ae4d00fe845121b P 28157bc3e678428fb5f93c077d528513: Bootstrap complete.
I20240502 01:55:38.763944 4723 ts_tablet_manager.cc:1367] T 2036e577d94242728ae4d00fe845121b P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240502 01:55:38.764523 4723 raft_consensus.cc:348] T 2036e577d94242728ae4d00fe845121b P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:38.764739 4723 raft_consensus.cc:729] T 2036e577d94242728ae4d00fe845121b P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:55:38.765019 4723 consensus_queue.cc:260] T 2036e577d94242728ae4d00fe845121b P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:38.767009 4723 ts_tablet_manager.cc:1398] T 2036e577d94242728ae4d00fe845121b P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:55:38.768601 24623 tablet_copy_service.cc:342] P 0c64d40b5d7644bd8b29feaa80eab72f: Request end of tablet copy session 28157bc3e678428fb5f93c077d528513-2036e577d94242728ae4d00fe845121b received from {username='slave'} at 127.14.212.198:35347
I20240502 01:55:38.768699 24623 tablet_copy_service.cc:433] P 0c64d40b5d7644bd8b29feaa80eab72f: ending tablet copy session 28157bc3e678428fb5f93c077d528513-2036e577d94242728ae4d00fe845121b on tablet 2036e577d94242728ae4d00fe845121b with peer 28157bc3e678428fb5f93c077d528513
I20240502 01:55:38.793501 21988 raft_consensus.cc:1203] T feffa925599b4d6aafa48e09e4f6467a P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:38.837388 4688 ts_tablet_manager.cc:902] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 06543b05a16f42099bc86db2a0284412: Initiating tablet copy from peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:55:38.839747 4688 tablet_copy_client.cc:308] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 06543b05a16f42099bc86db2a0284412: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.197:36677
I20240502 01:55:38.840391 24623 tablet_copy_service.cc:140] P 0c64d40b5d7644bd8b29feaa80eab72f: Received BeginTabletCopySession request for tablet 2b0e8db8c3414ce0a5ff12200fe6b544 from peer 06543b05a16f42099bc86db2a0284412 ({username='slave'} at 127.14.212.194:42615)
I20240502 01:55:38.840561 24623 tablet_copy_service.cc:161] P 0c64d40b5d7644bd8b29feaa80eab72f: Beginning new tablet copy session on tablet 2b0e8db8c3414ce0a5ff12200fe6b544 from peer 06543b05a16f42099bc86db2a0284412 at {username='slave'} at 127.14.212.194:42615: session id = 06543b05a16f42099bc86db2a0284412-2b0e8db8c3414ce0a5ff12200fe6b544
I20240502 01:55:38.843379 24623 tablet_copy_source_session.cc:205] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:38.844606 4688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b0e8db8c3414ce0a5ff12200fe6b544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:38.851261 4688 tablet_copy_client.cc:783] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 06543b05a16f42099bc86db2a0284412: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:38.851505 4688 tablet_copy_client.cc:647] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 06543b05a16f42099bc86db2a0284412: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:38.853209 4688 tablet_copy_client.cc:523] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 06543b05a16f42099bc86db2a0284412: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:38.857136 4688 tablet_bootstrap.cc:492] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 06543b05a16f42099bc86db2a0284412: Bootstrap starting.
I20240502 01:55:38.862268 4688 tablet_bootstrap.cc:492] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 06543b05a16f42099bc86db2a0284412: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:38.862576 4688 tablet_bootstrap.cc:492] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 06543b05a16f42099bc86db2a0284412: Bootstrap complete.
I20240502 01:55:38.862808 4688 ts_tablet_manager.cc:1367] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 06543b05a16f42099bc86db2a0284412: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240502 01:55:38.863404 4688 raft_consensus.cc:348] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:38.863574 4688 raft_consensus.cc:729] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Initialized, Role: FOLLOWER
I20240502 01:55:38.863762 4688 consensus_queue.cc:260] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 06543b05a16f42099bc86db2a0284412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:38.878270 4688 ts_tablet_manager.cc:1398] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 06543b05a16f42099bc86db2a0284412: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20240502 01:55:38.879146 24623 tablet_copy_service.cc:342] P 0c64d40b5d7644bd8b29feaa80eab72f: Request end of tablet copy session 06543b05a16f42099bc86db2a0284412-2b0e8db8c3414ce0a5ff12200fe6b544 received from {username='slave'} at 127.14.212.194:42615
I20240502 01:55:38.879290 24623 tablet_copy_service.cc:433] P 0c64d40b5d7644bd8b29feaa80eab72f: ending tablet copy session 06543b05a16f42099bc86db2a0284412-2b0e8db8c3414ce0a5ff12200fe6b544 on tablet 2b0e8db8c3414ce0a5ff12200fe6b544 with peer 06543b05a16f42099bc86db2a0284412
I20240502 01:55:38.903976 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling FlushMRSOp(766241df5ee24264a5cb7176e6c97b91): perf score=0.033361
I20240502 01:55:38.960954 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling FlushMRSOp(766241df5ee24264a5cb7176e6c97b91): perf score=0.033379
I20240502 01:55:38.994638 4785 ts_tablet_manager.cc:902] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f: Initiating tablet copy from peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:55:38.999488 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: FlushMRSOp(766241df5ee24264a5cb7176e6c97b91) complete. Timing: real 0.095s user 0.055s sys 0.021s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6372,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":7451,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":817,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":33990,"threads_started":2,"wal-append.queue_time_us":27026}
I20240502 01:55:39.002183 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling UndoDeltaBlockGCOp(766241df5ee24264a5cb7176e6c97b91): 4834 bytes on disk
I20240502 01:55:39.002559 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling FlushMRSOp(3a27dc6af117475cb227e3526d2a9007): perf score=0.033392
I20240502 01:55:39.002887 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: UndoDeltaBlockGCOp(766241df5ee24264a5cb7176e6c97b91) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
I20240502 01:55:39.005024 4785 tablet_copy_client.cc:308] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.193:33785
I20240502 01:55:39.005834 21878 tablet_copy_service.cc:140] P 759257ea7bef415ca698e60d82199d77: Received BeginTabletCopySession request for tablet 72f46ec6ed7848438cb9ad6e38053c4d from peer 0c64d40b5d7644bd8b29feaa80eab72f ({username='slave'} at 127.14.212.197:36517)
I20240502 01:55:39.005990 21878 tablet_copy_service.cc:161] P 759257ea7bef415ca698e60d82199d77: Beginning new tablet copy session on tablet 72f46ec6ed7848438cb9ad6e38053c4d from peer 0c64d40b5d7644bd8b29feaa80eab72f at {username='slave'} at 127.14.212.197:36517: session id = 0c64d40b5d7644bd8b29feaa80eab72f-72f46ec6ed7848438cb9ad6e38053c4d
I20240502 01:55:39.009907 21878 tablet_copy_source_session.cc:205] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:39.011673 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f46ec6ed7848438cb9ad6e38053c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:39.018410 4785 tablet_copy_client.cc:783] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:39.018729 4785 tablet_copy_client.cc:647] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:39.021170 4785 tablet_copy_client.cc:523] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:39.025645 4785 tablet_bootstrap.cc:492] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
W20240502 01:55:39.042683 22054 connection.cc:535] client connection to 127.14.212.199:35737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:39.042971 22054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:39.042982 21791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.212.199:35737 (error 108)
W20240502 01:55:39.043537 21923 connection.cc:535] client connection to 127.14.212.199:35737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:39.043845 21923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:39.044860 22185 connection.cc:535] client connection to 127.14.212.199:35737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:39.045176 22185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:39.045711 24934 connection.cc:535] client connection to 127.14.212.199:35737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:39.046016 24934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:39.047827 24537 connection.cc:535] client connection to 127.14.212.199:35737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:39.048130 24537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:39.051666 24669 connection.cc:535] client connection to 127.14.212.199:35737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:39.051968 24669 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:55:39.058825 24735 raft_consensus.cc:1203] T f018db81d6214bd0ac12feaf54541fed P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:39.084573 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: FlushMRSOp(766241df5ee24264a5cb7176e6c97b91) complete. Timing: real 0.123s user 0.081s sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6359,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":12257,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":981,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":13070,"threads_started":2,"wal-append.queue_time_us":6429}
I20240502 01:55:39.086917 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling UndoDeltaBlockGCOp(766241df5ee24264a5cb7176e6c97b91): 4834 bytes on disk
I20240502 01:55:39.087779 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: UndoDeltaBlockGCOp(766241df5ee24264a5cb7176e6c97b91) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":161,"lbm_reads_lt_1ms":4}
I20240502 01:55:39.088968 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling FlushMRSOp(0d246b118c1c4388ac04b3775ddbee89): perf score=0.033400
I20240502 01:55:39.089907 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling FlushMRSOp(3a27dc6af117475cb227e3526d2a9007): perf score=0.033407
I20240502 01:55:39.181165 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: FlushMRSOp(3a27dc6af117475cb227e3526d2a9007) complete. Timing: real 0.083s user 0.044s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1615,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":975,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":8416,"threads_started":1,"wal-append.queue_time_us":8812}
I20240502 01:55:39.183758 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling UndoDeltaBlockGCOp(3a27dc6af117475cb227e3526d2a9007): 4581 bytes on disk
I20240502 01:55:39.184595 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: FlushMRSOp(0d246b118c1c4388ac04b3775ddbee89) complete. Timing: real 0.095s user 0.046s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3524,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1467,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1161,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":8505,"threads_started":2,"wal-append.queue_time_us":3862}
I20240502 01:55:39.191638 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling UndoDeltaBlockGCOp(0d246b118c1c4388ac04b3775ddbee89): 4921 bytes on disk
I20240502 01:55:39.192287 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: UndoDeltaBlockGCOp(0d246b118c1c4388ac04b3775ddbee89) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240502 01:55:39.197705 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: UndoDeltaBlockGCOp(3a27dc6af117475cb227e3526d2a9007) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":177,"lbm_reads_lt_1ms":4}
I20240502 01:55:39.257251 4801 ts_tablet_manager.cc:902] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 759257ea7bef415ca698e60d82199d77: Initiating tablet copy from peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
W20240502 01:55:39.261751 24669 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.199:35737, user_credentials={real_user=slave}} blocked reactor thread for 209835us
W20240502 01:55:39.265517 24537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.199:35737, user_credentials={real_user=slave}} blocked reactor thread for 217432us
W20240502 01:55:39.266698 21923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.199:35737, user_credentials={real_user=slave}} blocked reactor thread for 222906us
W20240502 01:55:39.271193 22185 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.199:35737, user_credentials={real_user=slave}} blocked reactor thread for 226066us
W20240502 01:55:39.273988 22054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.199:35737, user_credentials={real_user=slave}} blocked reactor thread for 231060us
W20240502 01:55:39.282370 24539 connection.cc:441] server connection from 127.14.212.199:34671 torn down before Call kudu.consensus.ConsensusService.StartTabletCopy from 127.14.212.199:34671 (request call id 1229) could send its response
W20240502 01:55:39.295118 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.295328 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.295413 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.295637 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.295787 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.295899 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.345671 21791 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.14.212.199:35737, user_credentials={real_user=slave}} blocked reactor thread for 302741us
I20240502 01:55:39.349762 22118 raft_consensus.cc:1203] T ad8bc5be633d4646a7c10d2483ed4e5b P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 01:55:39.351069 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.351254 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.351352 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.351882 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.352005 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.352077 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.352130 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.365226 24669 consensus_peers.cc:485] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.365424 24669 consensus_peers.cc:485] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.365535 24669 consensus_peers.cc:485] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.366070 24669 consensus_peers.cc:485] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.372249 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.372442 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.372697 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.386082 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:39.395987 24669 consensus_peers.cc:485] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.398855 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.409168 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:39.417171 4801 tablet_copy_client.cc:308] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 759257ea7bef415ca698e60d82199d77: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.195:38879
I20240502 01:55:39.418330 22140 tablet_copy_service.cc:140] P c64840657cdf4b93980bd34120c94d9a: Received BeginTabletCopySession request for tablet 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 from peer 759257ea7bef415ca698e60d82199d77 ({username='slave'} at 127.14.212.193:56135)
I20240502 01:55:39.418512 22140 tablet_copy_service.cc:161] P c64840657cdf4b93980bd34120c94d9a: Beginning new tablet copy session on tablet 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 from peer 759257ea7bef415ca698e60d82199d77 at {username='slave'} at 127.14.212.193:56135: session id = 759257ea7bef415ca698e60d82199d77-28dbb8d9ef5f4efdb5ba6e8ba9179aa8
I20240502 01:55:39.422000 22140 tablet_copy_source_session.cc:205] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 01:55:39.422796 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:39.424429 4801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28dbb8d9ef5f4efdb5ba6e8ba9179aa8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:55:39.429258 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.429558 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.429751 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.429921 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.436978 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.437317 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.437503 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.437690 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.437849 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.437999 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.438186 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.438364 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:39.445219 4801 tablet_copy_client.cc:783] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:39.445914 4801 tablet_copy_client.cc:647] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:39.448802 4801 tablet_copy_client.cc:523] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 759257ea7bef415ca698e60d82199d77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:39.457573 4801 tablet_bootstrap.cc:492] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:55:39.467217 24735 raft_consensus.cc:1203] T 2036e577d94242728ae4d00fe845121b P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 01:55:39.486110 22184 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.193:33785, user_credentials={real_user=slave}} blocked reactor thread for 72535us
W20240502 01:55:39.491685 21923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.197:36677, user_credentials={real_user=slave}} blocked reactor thread for 52458.4us
I20240502 01:55:39.491990 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: FlushMRSOp(3a27dc6af117475cb227e3526d2a9007) complete. Timing: real 0.489s user 0.109s sys 0.039s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":87379,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":13017,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":962,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":118973,"threads_started":2,"wal-append.queue_time_us":31837}
I20240502 01:55:39.494988 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling FlushMRSOp(0d246b118c1c4388ac04b3775ddbee89): perf score=0.033524
I20240502 01:55:39.500905 21985 raft_consensus.cc:1203] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:39.525507 4801 tablet_bootstrap.cc:492] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 759257ea7bef415ca698e60d82199d77: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:55:39.526724 4801 tablet_bootstrap.cc:492] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 759257ea7bef415ca698e60d82199d77: Bootstrap complete.
I20240502 01:55:39.527005 4801 ts_tablet_manager.cc:1367] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.070s user 0.007s sys 0.004s
I20240502 01:55:39.527884 4801 raft_consensus.cc:348] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 759257ea7bef415ca698e60d82199d77 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
W20240502 01:55:39.528101 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.528424 4801 raft_consensus.cc:729] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:55:39.528630 4801 consensus_queue.cc:260] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:39.539971 4816 raft_consensus.cc:482] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.540216 4816 raft_consensus.cc:504] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:39.540345 21854 raft_consensus.cc:1203] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:39.541159 4816 leader_election.cc:288] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:55:39.541807 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1803e35563e84c84b90cbf608ecf6a93" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:55:39.541977 4352 raft_consensus.cc:2932] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
W20240502 01:55:39.543325 24669 leader_election.cc:334] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.543447 24669 leader_election.cc:302] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28157bc3e678428fb5f93c077d528513; no voters: c2a28baa7ee2469294d94b7fe5455cff, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.543952 4816 raft_consensus.cc:2726] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:39.547431 21854 raft_consensus.cc:2932] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:39.553419 4566 raft_consensus.cc:482] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.553623 4566 raft_consensus.cc:504] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:39.554533 4566 leader_election.cc:288] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:55:39.554999 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c3475bcad94e3dad4be87376e500bc" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
W20240502 01:55:39.556552 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.556677 21791 leader_election.cc:302] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.557301 4566 raft_consensus.cc:2726] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:39.573570 4801 ts_tablet_manager.cc:1398] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240502 01:55:39.574132 24999 raft_consensus.cc:2932] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:39.578202 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: FlushMRSOp(0d246b118c1c4388ac04b3775ddbee89) complete. Timing: real 0.083s user 0.030s sys 0.033s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1408,"drs_written":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1053,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6866,"threads_started":1,"wal-append.queue_time_us":6734}
I20240502 01:55:39.580686 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling UndoDeltaBlockGCOp(0d246b118c1c4388ac04b3775ddbee89): 4921 bytes on disk
I20240502 01:55:39.580827 22140 tablet_copy_service.cc:342] P c64840657cdf4b93980bd34120c94d9a: Request end of tablet copy session 759257ea7bef415ca698e60d82199d77-28dbb8d9ef5f4efdb5ba6e8ba9179aa8 received from {username='slave'} at 127.14.212.193:56135
I20240502 01:55:39.580966 22140 tablet_copy_service.cc:433] P c64840657cdf4b93980bd34120c94d9a: ending tablet copy session 759257ea7bef415ca698e60d82199d77-28dbb8d9ef5f4efdb5ba6e8ba9179aa8 on tablet 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 with peer 759257ea7bef415ca698e60d82199d77
I20240502 01:55:39.580989 21712 catalog_manager.cc:5484] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:39.581831 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: UndoDeltaBlockGCOp(0d246b118c1c4388ac04b3775ddbee89) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240502 01:55:39.583417 4819 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.583580 4819 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:39.583884 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling UndoDeltaBlockGCOp(3a27dc6af117475cb227e3526d2a9007): 4581 bytes on disk
I20240502 01:55:39.585378 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
W20240502 01:55:39.586391 22185 leader_election.cc:334] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.586594 22184 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 759257ea7bef415ca698e60d82199d77, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.586959 4647 raft_consensus.cc:2726] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:39.587802 4819 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
W20240502 01:55:39.589169 22185 leader_election.cc:341] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 36d358202b544e74bcc378060082df4d
I20240502 01:55:39.590930 4785 tablet_bootstrap.cc:492] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:39.591233 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: UndoDeltaBlockGCOp(3a27dc6af117475cb227e3526d2a9007) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240502 01:55:39.591383 4785 tablet_bootstrap.cc:492] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap complete.
I20240502 01:55:39.593184 4819 raft_consensus.cc:482] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.594375 4136 raft_consensus.cc:482] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.594601 4136 raft_consensus.cc:504] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:39.593334 4819 raft_consensus.cc:504] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:39.595470 4136 leader_election.cc:288] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:55:39.596849 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c351a8e007d04327bb14303d7b32d5be" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
W20240502 01:55:39.597373 22185 leader_election.cc:334] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.598186 4785 ts_tablet_manager.cc:1367] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.573s user 0.150s sys 0.191s
I20240502 01:55:39.598234 4819 leader_election.cc:288] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:55:39.598431 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea30e2e1ef6473f880a001e57f27171" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
W20240502 01:55:39.598686 21923 leader_election.cc:334] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.598832 21923 leader_election.cc:302] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06543b05a16f42099bc86db2a0284412; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.599120 4136 raft_consensus.cc:2726] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:39.599009 4785 raft_consensus.cc:348] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:39.599117 22186 leader_election.cc:302] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: c64840657cdf4b93980bd34120c94d9a, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.599239 4785 raft_consensus.cc:729] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:55:39.599438 4814 raft_consensus.cc:2726] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:39.599499 4785 consensus_queue.cc:260] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:39.601594 4785 ts_tablet_manager.cc:1398] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:55:39.602355 21878 tablet_copy_service.cc:342] P 759257ea7bef415ca698e60d82199d77: Request end of tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-72f46ec6ed7848438cb9ad6e38053c4d received from {username='slave'} at 127.14.212.197:36517
I20240502 01:55:39.602499 21878 tablet_copy_service.cc:433] P 759257ea7bef415ca698e60d82199d77: ending tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-72f46ec6ed7848438cb9ad6e38053c4d on tablet 72f46ec6ed7848438cb9ad6e38053c4d with peer 0c64d40b5d7644bd8b29feaa80eab72f
I20240502 01:55:39.637190 4352 raft_consensus.cc:482] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.637388 4352 raft_consensus.cc:504] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:39.638113 4352 leader_election.cc:288] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:39.639750 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a90f08d29a49d3b451f85cbdc828a6" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:55:39.644009 4352 raft_consensus.cc:482] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.644168 4352 raft_consensus.cc:504] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:39.644755 4352 leader_election.cc:288] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:55:39.650256 22054 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.650465 22054 leader_election.cc:302] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.650457 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60b948dae4841dd9c0919f2fcac5299" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:55:39.651437 4352 raft_consensus.cc:2726] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:55:39.652648 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.652770 22054 leader_election.cc:302] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.653306 4352 raft_consensus.cc:2726] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:39.654186 4816 raft_consensus.cc:482] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.654381 4816 raft_consensus.cc:504] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:39.655226 4816 leader_election.cc:288] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
W20240502 01:55:39.657698 24669 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.658648 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60b948dae4841dd9c0919f2fcac5299" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:55:39.658910 22120 raft_consensus.cc:2454] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:55:39.660017 24670 leader_election.cc:302] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, c64840657cdf4b93980bd34120c94d9a; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.660485 4816 raft_consensus.cc:2781] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:39.660583 4816 raft_consensus.cc:482] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.660679 4816 raft_consensus.cc:3037] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:39.663998 4816 raft_consensus.cc:504] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:39.665148 4816 leader_election.cc:288] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:55:39.665427 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60b948dae4841dd9c0919f2fcac5299" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:55:39.665635 22120 raft_consensus.cc:3037] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:55:39.666409 24669 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.668336 22120 raft_consensus.cc:2454] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 2.
I20240502 01:55:39.669486 24670 leader_election.cc:302] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, c64840657cdf4b93980bd34120c94d9a; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.669971 4476 raft_consensus.cc:2781] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:39.670145 4476 raft_consensus.cc:686] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [term 2 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:55:39.670431 4476 consensus_queue.cc:237] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:39.679936 21707 catalog_manager.cc:5484] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 1 to 2, leader changed from d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 2 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:39.688073 4819 raft_consensus.cc:482] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.688253 4819 raft_consensus.cc:504] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:39.689075 4819 leader_election.cc:288] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:39.690281 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1803e35563e84c84b90cbf608ecf6a93" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:55:39.690632 24735 raft_consensus.cc:2454] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:55:39.691273 22185 leader_election.cc:302] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:55:39.691628 4819 raft_consensus.cc:2781] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:39.691730 4819 raft_consensus.cc:482] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.691800 4819 raft_consensus.cc:3037] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:55:39.691627 22185 leader_election.cc:334] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.699539 4819 raft_consensus.cc:504] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
W20240502 01:55:39.701692 22185 leader_election.cc:334] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.702543 4819 leader_election.cc:288] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:39.702868 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1803e35563e84c84b90cbf608ecf6a93" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:55:39.703114 24735 raft_consensus.cc:3037] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:39.704331 22251 consensus_queue.cc:237] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:39.705371 24735 raft_consensus.cc:2454] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 2.
I20240502 01:55:39.705873 22185 leader_election.cc:302] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, c2a28baa7ee2469294d94b7fe5455cff; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.706332 4819 raft_consensus.cc:2781] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:39.706472 4819 raft_consensus.cc:686] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [term 2 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:55:39.706998 4819 consensus_queue.cc:237] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:39.708989 4814 raft_consensus.cc:482] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.709141 4814 raft_consensus.cc:504] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:39.709793 4814 leader_election.cc:288] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:55:39.711957 4784 raft_consensus.cc:482] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.712162 4784 raft_consensus.cc:504] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:39.712944 4784 leader_election.cc:288] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:55:39.713073 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3237002dce97460799e219cf5b4968cb" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
W20240502 01:55:39.713871 22185 leader_election.cc:334] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.714126 22185 leader_election.cc:302] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.714359 21708 catalog_manager.cc:5484] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 1 to 2, leader changed from d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 2 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:39.714613 21985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c351a8e007d04327bb14303d7b32d5be" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:55:39.714815 21985 raft_consensus.cc:2454] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:55:39.715286 24536 leader_election.cc:302] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 0c64d40b5d7644bd8b29feaa80eab72f; no voters:
W20240502 01:55:39.715448 24537 leader_election.cc:334] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.715581 4784 raft_consensus.cc:2781] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:39.715678 4784 raft_consensus.cc:482] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.715785 4784 raft_consensus.cc:3037] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:39.718252 4819 raft_consensus.cc:2726] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:39.718820 4784 raft_consensus.cc:504] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:39.719590 4784 leader_election.cc:288] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:55:39.720054 21985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c351a8e007d04327bb14303d7b32d5be" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:55:39.720237 21985 raft_consensus.cc:3037] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:55:39.721223 24537 leader_election.cc:334] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.723217 21985 raft_consensus.cc:2454] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 2.
I20240502 01:55:39.723734 24536 leader_election.cc:302] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 0c64d40b5d7644bd8b29feaa80eab72f; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.724009 4784 raft_consensus.cc:2781] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:39.724231 4784 raft_consensus.cc:686] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:55:39.724640 4784 consensus_queue.cc:237] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:39.726289 24735 raft_consensus.cc:1261] T 41e68a88b92347e8855b6e5c82e7b7e1 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:55:39.726528 22186 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41e68a88b92347e8855b6e5c82e7b7e1. This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:39.727185 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:39.727248 4814 consensus_queue.cc:1035] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:39.730293 21708 catalog_manager.cc:5484] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 1 to 2, leader changed from d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 2 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:39.733294 4814 raft_consensus.cc:482] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.733466 4814 raft_consensus.cc:504] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:39.734185 4814 leader_election.cc:288] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:55:39.734985 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce01a3c9d9948ba8f153ad150cf3cb0" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
W20240502 01:55:39.735402 22185 leader_election.cc:334] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.735553 22186 leader_election.cc:302] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: c64840657cdf4b93980bd34120c94d9a, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.735816 4814 raft_consensus.cc:2726] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:39.743263 4476 raft_consensus.cc:482] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.743450 4476 raft_consensus.cc:504] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:39.744140 4476 leader_election.cc:288] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:55:39.745025 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59056268198e4b8f9f4d2afe1132f9c8" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
W20240502 01:55:39.745932 24669 leader_election.cc:334] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.746064 24669 leader_election.cc:302] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28157bc3e678428fb5f93c077d528513; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.746328 4476 raft_consensus.cc:2726] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:39.751982 4817 raft_consensus.cc:482] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.752158 4817 raft_consensus.cc:504] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:39.752813 4817 leader_election.cc:288] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:55:39.753436 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564f981f7f7c4f2480c4fd1e5b5c1ad8" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
W20240502 01:55:39.754206 24934 leader_election.cc:334] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.754331 24934 leader_election.cc:302] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.754750 4817 raft_consensus.cc:2726] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:39.755787 4784 raft_consensus.cc:482] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.755966 4784 raft_consensus.cc:504] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:39.756767 4784 leader_election.cc:288] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:55:39.757118 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a16aff54e6f4975a97776c214890f57" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:55:39.757934 4817 raft_consensus.cc:482] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.758121 4817 raft_consensus.cc:504] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:39.758340 4352 raft_consensus.cc:482] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.758500 4352 raft_consensus.cc:504] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:39.758795 4817 leader_election.cc:288] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:55:39.759135 4352 leader_election.cc:288] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
W20240502 01:55:39.759631 24537 leader_election.cc:334] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.759702 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "641ed1f2bf7e40d7a8a6472cf9cd5519" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:55:39.759788 24537 leader_election.cc:302] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 759257ea7bef415ca698e60d82199d77, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.759737 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce01a3c9d9948ba8f153ad150cf3cb0" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:55:39.759981 22251 raft_consensus.cc:2454] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:55:39.760433 22055 leader_election.cc:302] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters:
W20240502 01:55:39.760617 22054 leader_election.cc:334] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.760793 4352 raft_consensus.cc:2781] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:39.760959 4352 raft_consensus.cc:482] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.761061 4352 raft_consensus.cc:3037] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:39.761238 4784 raft_consensus.cc:2726] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:55:39.761808 24934 leader_election.cc:334] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.761960 24934 leader_election.cc:302] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 759257ea7bef415ca698e60d82199d77, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.762274 4817 raft_consensus.cc:2726] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:39.763481 4352 raft_consensus.cc:504] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:39.763998 4352 leader_election.cc:288] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:55:39.764546 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce01a3c9d9948ba8f153ad150cf3cb0" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:55:39.764783 22251 raft_consensus.cc:3037] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:39.767642 22251 raft_consensus.cc:2454] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 2.
W20240502 01:55:39.767710 22054 leader_election.cc:334] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.768024 22055 leader_election.cc:302] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.768301 4352 raft_consensus.cc:2781] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:39.768476 4352 raft_consensus.cc:686] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [term 2 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:55:39.768841 4352 consensus_queue.cc:237] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:39.772477 21708 catalog_manager.cc:5484] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 1 to 2, leader changed from d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 2 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:39.783794 4817 raft_consensus.cc:482] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.783957 4817 raft_consensus.cc:504] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:39.784241 4820 raft_consensus.cc:482] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.784391 4820 raft_consensus.cc:504] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:39.784915 4817 leader_election.cc:288] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:55:39.785209 4820 leader_election.cc:288] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:39.785457 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458b19e5186f4ce79f618a10e101c605" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:55:39.786127 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f537605bc746a0807e3c0a03ff2cc3" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
W20240502 01:55:39.786823 21923 leader_election.cc:334] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.787081 21923 leader_election.cc:302] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06543b05a16f42099bc86db2a0284412; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.787308 4820 raft_consensus.cc:2726] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:55:39.787652 24934 leader_election.cc:334] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.787788 24934 leader_election.cc:302] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: c2a28baa7ee2469294d94b7fe5455cff, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.788044 4817 raft_consensus.cc:2726] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:39.819653 4784 raft_consensus.cc:482] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.819882 4784 raft_consensus.cc:504] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:39.820782 4784 leader_election.cc:288] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:55:39.821156 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3237002dce97460799e219cf5b4968cb" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:55:39.821424 22251 raft_consensus.cc:2454] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:55:39.821866 24538 leader_election.cc:302] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:55:39.822232 4821 raft_consensus.cc:2781] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:39.822327 4821 raft_consensus.cc:482] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.822402 4821 raft_consensus.cc:3037] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:55:39.822639 24537 leader_election.cc:334] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.824657 4821 raft_consensus.cc:504] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:39.825336 4821 leader_election.cc:288] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:55:39.825898 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3237002dce97460799e219cf5b4968cb" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:55:39.826198 22251 raft_consensus.cc:3037] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:55:39.826992 24537 leader_election.cc:334] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.829308 22251 raft_consensus.cc:2454] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 2.
I20240502 01:55:39.829728 24538 leader_election.cc:302] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, c2a28baa7ee2469294d94b7fe5455cff; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.830042 4821 raft_consensus.cc:2781] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:39.830202 4821 raft_consensus.cc:686] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:55:39.830472 4821 consensus_queue.cc:237] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:39.834954 21708 catalog_manager.cc:5484] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 1 to 2, leader changed from d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 2 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:39.836974 4784 raft_consensus.cc:482] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.837170 4784 raft_consensus.cc:504] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:39.837877 4784 leader_election.cc:288] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:39.838989 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a4a927b8f1740208f99ab1aca81b67a" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
W20240502 01:55:39.839165 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.839191 4819 raft_consensus.cc:482] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.839375 4819 raft_consensus.cc:504] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:39.839577 24537 leader_election.cc:302] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.839987 4784 raft_consensus.cc:2726] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:39.840080 4819 leader_election.cc:288] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:55:39.840569 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458b19e5186f4ce79f618a10e101c605" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:55:39.840821 25000 raft_consensus.cc:2454] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:55:39.841289 22187 leader_election.cc:302] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, c2a28baa7ee2469294d94b7fe5455cff; no voters:
W20240502 01:55:39.841529 22185 leader_election.cc:334] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.841810 4819 raft_consensus.cc:2781] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:39.841902 4819 raft_consensus.cc:482] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.842000 4819 raft_consensus.cc:3037] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:39.844226 4819 raft_consensus.cc:504] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:39.847863 4476 raft_consensus.cc:482] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.848066 4476 raft_consensus.cc:504] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:39.848827 4476 leader_election.cc:288] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:55:39.849951 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a90f08d29a49d3b451f85cbdc828a6" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:55:39.850153 22120 raft_consensus.cc:2454] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:55:39.850543 24670 leader_election.cc:302] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, c64840657cdf4b93980bd34120c94d9a; no voters:
W20240502 01:55:39.850749 24669 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.850833 4476 raft_consensus.cc:2781] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:39.850947 4476 raft_consensus.cc:482] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.851070 4476 raft_consensus.cc:3037] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:39.853096 4476 raft_consensus.cc:504] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:39.853672 4476 leader_election.cc:288] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 election: Requested vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:55:39.854075 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a90f08d29a49d3b451f85cbdc828a6" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a"
I20240502 01:55:39.854313 22120 raft_consensus.cc:3037] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:55:39.855161 24669 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.856812 22120 raft_consensus.cc:2454] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 2.
I20240502 01:55:39.857146 3952 raft_consensus.cc:482] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.857344 3952 raft_consensus.cc:504] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:39.857983 3952 leader_election.cc:288] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:55:39.858284 24670 leader_election.cc:302] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, c64840657cdf4b93980bd34120c94d9a; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.858824 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a16aff54e6f4975a97776c214890f57" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:55:39.859076 24603 raft_consensus.cc:2454] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:55:39.859840 4476 raft_consensus.cc:2781] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:39.860008 4476 raft_consensus.cc:686] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [term 2 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:55:39.860288 4476 consensus_queue.cc:237] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
W20240502 01:55:39.860674 21791 leader_election.cc:334] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.861001 21791 leader_election.cc:302] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 759257ea7bef415ca698e60d82199d77; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.861255 3952 raft_consensus.cc:2781] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:39.861332 3952 raft_consensus.cc:482] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.861404 3952 raft_consensus.cc:3037] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:39.863941 3952 raft_consensus.cc:504] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:39.864603 3952 leader_election.cc:288] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:55:39.865013 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a16aff54e6f4975a97776c214890f57" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:55:39.865249 24603 raft_consensus.cc:3037] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:55:39.867214 21791 leader_election.cc:334] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.867974 24603 raft_consensus.cc:2454] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 2.
I20240502 01:55:39.868357 21791 leader_election.cc:302] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 759257ea7bef415ca698e60d82199d77; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.868945 3952 raft_consensus.cc:2781] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:39.869081 3952 raft_consensus.cc:686] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [term 2 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:55:39.869396 3952 consensus_queue.cc:237] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:39.878441 4838 raft_consensus.cc:482] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.878664 4838 raft_consensus.cc:504] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:39.879477 4838 leader_election.cc:288] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:55:39.880555 21985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f537605bc746a0807e3c0a03ff2cc3" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:55:39.880792 21985 raft_consensus.cc:2454] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:55:39.881325 24668 leader_election.cc:302] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:55:39.882058 21708 catalog_manager.cc:5484] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 1 to 2, leader changed from d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 2 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:39.883154 4476 raft_consensus.cc:2781] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:39.883261 4476 raft_consensus.cc:482] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.883430 4476 raft_consensus.cc:3037] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:39.883268 21707 catalog_manager.cc:5484] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 1 to 2, leader changed from d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 2 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:39.886660 4476 raft_consensus.cc:504] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
W20240502 01:55:39.886912 24669 leader_election.cc:334] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.887404 4476 leader_election.cc:288] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:55:39.887964 21985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f537605bc746a0807e3c0a03ff2cc3" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:55:39.888216 21985 raft_consensus.cc:3037] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:39.889716 4856 raft_consensus.cc:482] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.889914 4856 raft_consensus.cc:504] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
W20240502 01:55:39.890441 24669 leader_election.cc:334] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.892899 21985 raft_consensus.cc:2454] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 2.
I20240502 01:55:39.893342 24668 leader_election.cc:302] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 28157bc3e678428fb5f93c077d528513; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.893286 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458b19e5186f4ce79f618a10e101c605" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:55:39.893584 25000 raft_consensus.cc:3037] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:39.893654 4476 raft_consensus.cc:2781] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:39.893798 4476 raft_consensus.cc:686] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [term 2 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:55:39.894073 4476 consensus_queue.cc:237] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:39.896759 25000 raft_consensus.cc:2454] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 2.
W20240502 01:55:39.898242 22185 leader_election.cc:334] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.898778 22187 leader_election.cc:302] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, c2a28baa7ee2469294d94b7fe5455cff; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.899068 4814 raft_consensus.cc:2781] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:39.899174 4814 raft_consensus.cc:686] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [term 2 LEADER]: Becoming Leader. State: Replica: c2a28baa7ee2469294d94b7fe5455cff, State: Running, Role: LEADER
I20240502 01:55:39.899400 4814 consensus_queue.cc:237] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:39.900624 4838 raft_consensus.cc:482] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.900811 4838 raft_consensus.cc:504] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:39.901809 4838 leader_election.cc:288] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
W20240502 01:55:39.904420 24669 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.905202 4856 leader_election.cc:288] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:55:39.910149 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c3475bcad94e3dad4be87376e500bc" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:55:39.910385 21857 raft_consensus.cc:2454] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:55:39.911818 4819 leader_election.cc:288] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:55:39.914335 24668 leader_election.cc:302] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 759257ea7bef415ca698e60d82199d77; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.914923 4838 raft_consensus.cc:2781] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:39.915092 4838 raft_consensus.cc:482] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.916940 4821 raft_consensus.cc:482] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.917094 4821 raft_consensus.cc:504] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:39.917732 4821 leader_election.cc:288] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:39.916718 4838 raft_consensus.cc:3037] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:39.920051 21708 catalog_manager.cc:5484] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: term changed from 1 to 2, leader changed from d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) to c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196). New cstate: current_term: 2 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:39.931046 4767 raft_consensus.cc:482] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.931252 4767 raft_consensus.cc:504] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:39.932076 4767 leader_election.cc:288] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:55:39.932590 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea30e2e1ef6473f880a001e57f27171" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
W20240502 01:55:39.933791 22054 leader_election.cc:334] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.932818 22251 raft_consensus.cc:2454] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:55:39.934516 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59056268198e4b8f9f4d2afe1132f9c8" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513" is_pre_election: true
I20240502 01:55:39.934648 22055 leader_election.cc:302] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.934798 24735 raft_consensus.cc:2454] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:55:39.935250 4767 raft_consensus.cc:2781] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:39.935449 4767 raft_consensus.cc:482] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.935575 4767 raft_consensus.cc:3037] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:39.938716 4767 raft_consensus.cc:504] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
W20240502 01:55:39.941159 22054 leader_election.cc:334] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.945974 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "641ed1f2bf7e40d7a8a6472cf9cd5519" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:55:39.946228 24999 raft_consensus.cc:2454] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:55:39.952968 21793 leader_election.cc:302] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:55:39.953450 4566 raft_consensus.cc:2781] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:39.953547 4566 raft_consensus.cc:482] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.953616 4566 raft_consensus.cc:3037] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:39.956754 4566 raft_consensus.cc:504] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:39.959803 4566 leader_election.cc:288] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:55:39.923491 4838 raft_consensus.cc:504] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:39.961823 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "641ed1f2bf7e40d7a8a6472cf9cd5519" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:55:39.962049 25000 raft_consensus.cc:3037] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:39.963747 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea30e2e1ef6473f880a001e57f27171" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:55:39.964017 22251 raft_consensus.cc:3037] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:39.965327 25000 raft_consensus.cc:2454] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 2.
I20240502 01:55:39.965338 4767 leader_election.cc:288] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:55:39.967856 22251 raft_consensus.cc:2454] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 2.
I20240502 01:55:39.967923 21793 leader_election.cc:302] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77; no voters:
I20240502 01:55:39.970245 4566 raft_consensus.cc:2781] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:39.970463 4566 raft_consensus.cc:686] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [term 2 LEADER]: Becoming Leader. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Running, Role: LEADER
I20240502 01:55:39.970703 4566 consensus_queue.cc:237] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:39.972432 22055 leader_election.cc:302] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.973687 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c3475bcad94e3dad4be87376e500bc" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:55:39.973866 21854 raft_consensus.cc:3037] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:55:39.974018 24669 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.974347 4352 raft_consensus.cc:2781] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:39.974552 4352 raft_consensus.cc:686] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [term 2 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:55:39.974897 4352 consensus_queue.cc:237] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:39.976725 21854 raft_consensus.cc:2454] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 2.
W20240502 01:55:39.977705 21791 leader_election.cc:334] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.978649 4838 leader_election.cc:288] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:55:39.980788 24537 leader_election.cc:302] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters:
W20240502 01:55:39.981215 24537 leader_election.cc:334] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:39.981418 4500 raft_consensus.cc:2781] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:39.981591 4500 raft_consensus.cc:482] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:39.981700 4500 raft_consensus.cc:3037] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:39.984580 4500 raft_consensus.cc:504] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:39.985462 4500 leader_election.cc:288] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: Requested vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:39.989522 24668 leader_election.cc:302] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28157bc3e678428fb5f93c077d528513, 759257ea7bef415ca698e60d82199d77; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:39.990100 4838 raft_consensus.cc:2781] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:39.990550 4838 raft_consensus.cc:686] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [term 2 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:55:39.990880 4838 consensus_queue.cc:237] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:39.990880 21708 catalog_manager.cc:5484] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 1 to 2, leader changed from d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 2 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } }
I20240502 01:55:39.996729 24603 raft_consensus.cc:1203] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 01:55:39.996809 21791 leader_election.cc:334] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:39.999342 24537 leader_election.cc:334] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:40.004144 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59056268198e4b8f9f4d2afe1132f9c8" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "28157bc3e678428fb5f93c077d528513"
I20240502 01:55:40.004460 24735 raft_consensus.cc:3037] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:40.004267 21708 catalog_manager.cc:5484] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 reported cstate change: term changed from 1 to 2, leader changed from d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) to 759257ea7bef415ca698e60d82199d77 (127.14.212.193). New cstate: current_term: 2 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
W20240502 01:55:40.004861 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:40.013283 24735 raft_consensus.cc:2454] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 2.
I20240502 01:55:40.013916 24537 leader_election.cc:302] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:40.013998 21707 catalog_manager.cc:5484] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 1 to 2, leader changed from d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 2 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:40.014329 4821 raft_consensus.cc:2781] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:40.014541 4821 raft_consensus.cc:686] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:55:40.014863 4821 consensus_queue.cc:237] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:39.996729 4784 raft_consensus.cc:482] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:40.021122 4784 raft_consensus.cc:504] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:40.024211 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564f981f7f7c4f2480c4fd1e5b5c1ad8" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:55:40.024473 25000 raft_consensus.cc:2454] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
W20240502 01:55:40.024470 24537 leader_election.cc:334] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:40.024739 4784 leader_election.cc:288] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:55:40.024995 24539 leader_election.cc:302] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 41c70f373c9d49a49891b03a12c6cb2b; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:40.025305 4821 raft_consensus.cc:2781] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:40.025420 4821 raft_consensus.cc:482] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:40.025525 4821 raft_consensus.cc:3037] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:40.027202 21708 catalog_manager.cc:5484] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 1 to 2, leader changed from d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 2 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:40.028234 4821 raft_consensus.cc:504] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:40.029282 4821 leader_election.cc:288] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:55:40.029660 21712 catalog_manager.cc:5484] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 1 to 2, leader changed from d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 2 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:40.030140 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564f981f7f7c4f2480c4fd1e5b5c1ad8" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:55:40.030392 25000 raft_consensus.cc:3037] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:55:40.030522 24537 leader_election.cc:334] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:40.032949 25000 raft_consensus.cc:2454] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 2.
I20240502 01:55:40.033489 24539 leader_election.cc:302] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 41c70f373c9d49a49891b03a12c6cb2b; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:40.033792 4821 raft_consensus.cc:2781] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:40.034248 4821 raft_consensus.cc:686] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:55:40.034535 4821 consensus_queue.cc:237] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:55:40.038179 21712 catalog_manager.cc:5484] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 1 to 2, leader changed from d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 2 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } }
I20240502 01:55:40.052974 3952 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:40.053165 3952 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:40.054208 3952 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:55:40.054908 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:55:40.055148 22249 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
W20240502 01:55:40.056404 21791 leader_election.cc:334] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:40.056665 21791 leader_election.cc:341] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 36d358202b544e74bcc378060082df4d
I20240502 01:55:40.056756 21791 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:40.057009 3952 raft_consensus.cc:2726] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:40.061445 4864 raft_consensus.cc:482] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:40.061621 4864 raft_consensus.cc:504] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:40.062183 4864 leader_election.cc:288] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:55:40.062628 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a4a927b8f1740208f99ab1aca81b67a" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:55:40.062857 24603 raft_consensus.cc:2454] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 1.
I20240502 01:55:40.063848 24669 leader_election.cc:302] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters:
I20240502 01:55:40.064154 4864 raft_consensus.cc:2781] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240502 01:55:40.064136 24669 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:40.064282 4864 raft_consensus.cc:482] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:40.064592 4864 raft_consensus.cc:3037] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:40.066498 4864 raft_consensus.cc:504] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:40.067070 4864 leader_election.cc:288] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:55:40.067528 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a4a927b8f1740208f99ab1aca81b67a" candidate_uuid: "28157bc3e678428fb5f93c077d528513" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:55:40.067745 24603 raft_consensus.cc:3037] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:55:40.068349 24669 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:40.069988 24603 raft_consensus.cc:2454] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28157bc3e678428fb5f93c077d528513 in term 2.
I20240502 01:55:40.070482 24669 leader_election.cc:302] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513; no voters: d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:40.070772 4864 raft_consensus.cc:2781] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:40.070927 4864 raft_consensus.cc:686] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [term 2 LEADER]: Becoming Leader. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Running, Role: LEADER
I20240502 01:55:40.071210 4864 consensus_queue.cc:237] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:40.076349 21712 catalog_manager.cc:5484] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 reported cstate change: term changed from 1 to 2, leader changed from d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199) to 28157bc3e678428fb5f93c077d528513 (127.14.212.198). New cstate: current_term: 2 leader_uuid: "28157bc3e678428fb5f93c077d528513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:40.120819 4868 ts_tablet_manager.cc:902] T 41e68a88b92347e8855b6e5c82e7b7e1 P c64840657cdf4b93980bd34120c94d9a: Initiating tablet copy from peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:55:40.122026 4868 tablet_copy_client.cc:308] T 41e68a88b92347e8855b6e5c82e7b7e1 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.196:43093
I20240502 01:55:40.122733 22271 tablet_copy_service.cc:140] P c2a28baa7ee2469294d94b7fe5455cff: Received BeginTabletCopySession request for tablet 41e68a88b92347e8855b6e5c82e7b7e1 from peer c64840657cdf4b93980bd34120c94d9a ({username='slave'} at 127.14.212.195:40447)
I20240502 01:55:40.122916 22271 tablet_copy_service.cc:161] P c2a28baa7ee2469294d94b7fe5455cff: Beginning new tablet copy session on tablet 41e68a88b92347e8855b6e5c82e7b7e1 from peer c64840657cdf4b93980bd34120c94d9a at {username='slave'} at 127.14.212.195:40447: session id = c64840657cdf4b93980bd34120c94d9a-41e68a88b92347e8855b6e5c82e7b7e1
I20240502 01:55:40.125387 22271 tablet_copy_source_session.cc:205] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:40.126755 4868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41e68a88b92347e8855b6e5c82e7b7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:40.132112 4868 tablet_copy_client.cc:783] T 41e68a88b92347e8855b6e5c82e7b7e1 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:40.132385 4868 tablet_copy_client.cc:647] T 41e68a88b92347e8855b6e5c82e7b7e1 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:40.134282 4868 tablet_copy_client.cc:523] T 41e68a88b92347e8855b6e5c82e7b7e1 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:40.137262 4868 tablet_bootstrap.cc:492] T 41e68a88b92347e8855b6e5c82e7b7e1 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:55:40.139236 24735 raft_consensus.cc:1261] T 1803e35563e84c84b90cbf608ecf6a93 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:55:40.140630 4819 consensus_queue.cc:1035] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:55:40.149411 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:40.155398 4868 tablet_bootstrap.cc:492] T 41e68a88b92347e8855b6e5c82e7b7e1 P c64840657cdf4b93980bd34120c94d9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
W20240502 01:55:40.155746 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:40.156082 4868 tablet_bootstrap.cc:492] T 41e68a88b92347e8855b6e5c82e7b7e1 P c64840657cdf4b93980bd34120c94d9a: Bootstrap complete.
I20240502 01:55:40.156435 4868 ts_tablet_manager.cc:1367] T 41e68a88b92347e8855b6e5c82e7b7e1 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.007s
I20240502 01:55:40.157392 4868 raft_consensus.cc:348] T 41e68a88b92347e8855b6e5c82e7b7e1 P c64840657cdf4b93980bd34120c94d9a [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:40.157959 4868 raft_consensus.cc:729] T 41e68a88b92347e8855b6e5c82e7b7e1 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:55:40.158187 4868 consensus_queue.cc:260] T 41e68a88b92347e8855b6e5c82e7b7e1 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:40.158758 4868 ts_tablet_manager.cc:1398] T 41e68a88b92347e8855b6e5c82e7b7e1 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:55:40.161800 22271 tablet_copy_service.cc:342] P c2a28baa7ee2469294d94b7fe5455cff: Request end of tablet copy session c64840657cdf4b93980bd34120c94d9a-41e68a88b92347e8855b6e5c82e7b7e1 received from {username='slave'} at 127.14.212.195:40447
I20240502 01:55:40.161924 22271 tablet_copy_service.cc:433] P c2a28baa7ee2469294d94b7fe5455cff: ending tablet copy session c64840657cdf4b93980bd34120c94d9a-41e68a88b92347e8855b6e5c82e7b7e1 on tablet 41e68a88b92347e8855b6e5c82e7b7e1 with peer c64840657cdf4b93980bd34120c94d9a
I20240502 01:55:40.221300 22249 raft_consensus.cc:1261] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:55:40.221884 4352 consensus_queue.cc:1035] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:55:40.226819 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:40.246744 24669 consensus_peers.cc:485] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:40.258767 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:40.279279 22120 raft_consensus.cc:1261] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:55:40.279433 22118 raft_consensus.cc:1261] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:55:40.280102 4476 consensus_queue.cc:1035] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:40.280033 4864 consensus_queue.cc:1035] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:55:40.286226 24669 consensus_peers.cc:485] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:40.294538 24669 consensus_peers.cc:485] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:40.330291 21985 raft_consensus.cc:1261] T c351a8e007d04327bb14303d7b32d5be P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:55:40.330999 4821 consensus_queue.cc:1035] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:55:40.336321 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:40.356124 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:40.401113 21985 raft_consensus.cc:1261] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:55:40.401727 4476 consensus_queue.cc:1035] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:40.424083 22251 raft_consensus.cc:1261] T 3237002dce97460799e219cf5b4968cb P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:55:40.424688 4821 consensus_queue.cc:1035] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:55:40.437039 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:40.442920 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:40.452834 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:40.457300 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:40.475273 24669 consensus_peers.cc:485] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:40.476979 24601 raft_consensus.cc:1261] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:55:40.477541 3952 consensus_queue.cc:1035] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:40.484757 25000 raft_consensus.cc:1261] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:55:40.485428 4819 consensus_queue.cc:1035] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:40.506844 21854 raft_consensus.cc:1261] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:55:40.507475 4838 consensus_queue.cc:1035] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:55:40.515089 24669 consensus_peers.cc:485] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:40.530392 22251 raft_consensus.cc:1261] T dea30e2e1ef6473f880a001e57f27171 P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:55:40.531014 4767 consensus_queue.cc:1035] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:55:40.552865 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:40.564626 24733 raft_consensus.cc:1261] T 59056268198e4b8f9f4d2afe1132f9c8 P 28157bc3e678428fb5f93c077d528513 [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:55:40.565356 4821 consensus_queue.cc:1035] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:40.577167 25000 raft_consensus.cc:1261] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:55:40.577729 4888 consensus_queue.cc:1035] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:40.602572 24601 raft_consensus.cc:1261] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Refusing update from remote peer 28157bc3e678428fb5f93c077d528513: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:55:40.603291 4476 consensus_queue.cc:1035] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:40.647531 25000 raft_consensus.cc:1261] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:55:40.648173 4500 consensus_queue.cc:1035] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:40.710898 22118 raft_consensus.cc:1203] T 41e68a88b92347e8855b6e5c82e7b7e1 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:40.711867 4890 raft_consensus.cc:2932] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:40.712611 22118 raft_consensus.cc:2932] T 41e68a88b92347e8855b6e5c82e7b7e1 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:40.715593 4890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:41996 (request call id 0) took 1012 ms. Trace:
I20240502 01:55:40.715759 4890 rpcz_store.cc:276] 0502 01:55:39.703466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 01:55:39.703595 (+ 129us) service_pool.cc:225] Handling call
0502 01:55:39.705260 (+ 1665us) log.cc:842] Serialized 505 byte log entry
0502 01:55:40.715536 (+1010276us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":332,"raft.queue_time_us":393,"raft.run_cpu_time_us":490,"raft.run_wall_time_us":490,"spinlock_wait_cycles":96640,"thread_start_us":19874,"threads_started":1,"wal-append.queue_time_us":19391}
I20240502 01:55:40.724829 21712 catalog_manager.cc:5484] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff reported cstate change: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:40.724826 24733 raft_consensus.cc:2932] T 41e68a88b92347e8855b6e5c82e7b7e1 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
W20240502 01:55:40.988796 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:41.188197 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:41.199507 24583 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:41.266314 24715 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 01:55:41.272804 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:55:41.276082 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:41.284207 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:41.287052 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:41.300145 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240502 01:55:41.365134 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.378001 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.383059 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.414625 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.429800 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.471597 24669 consensus_peers.cc:485] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.472805 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.519918 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.525012 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.527267 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.529825 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.533041 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:41.554014 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.581427 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.640050 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.643772 24669 consensus_peers.cc:485] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.655916 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.667162 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.669170 24669 consensus_peers.cc:485] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.669481 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.671408 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.677582 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.719894 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.744819 24669 consensus_peers.cc:485] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.763442 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.787312 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:55:41.811383 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:41.834208 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:55:41.850461 24583 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:41.871610 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:41.881481 24715 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 01:55:41.884222 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:55:41.887719 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240502 01:55:41.893424 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.894879 24669 consensus_peers.cc:485] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.905449 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:41.922509 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:55:41.940635 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:41.941488 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:55:41.950510 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 01:55:41.981480 22054 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.199:35737, user_credentials={real_user=slave}} blocked reactor thread for 59000.4us
W20240502 01:55:41.994211 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:42.053794 21792 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.195:38879, user_credentials={real_user=slave}} blocked reactor thread for 77692.3us
W20240502 01:55:42.065169 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:42.090698 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:55:42.120842 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling FlushMRSOp(ba39439749a9486799373e173110f5c5): perf score=0.033349
W20240502 01:55:42.160108 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:55:42.185019 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: FlushMRSOp(ba39439749a9486799373e173110f5c5) complete. Timing: real 0.064s user 0.038s sys 0.021s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":11380,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1778,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":703,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":17791,"threads_started":2,"wal-append.queue_time_us":6404}
I20240502 01:55:42.187985 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling UndoDeltaBlockGCOp(ba39439749a9486799373e173110f5c5): 4581 bytes on disk
I20240502 01:55:42.188939 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: UndoDeltaBlockGCOp(ba39439749a9486799373e173110f5c5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
I20240502 01:55:42.224593 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling FlushMRSOp(ec67f45b2b604c129d8572c5ca6acbc4): perf score=0.033378
I20240502 01:55:42.229349 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling FlushMRSOp(ba39439749a9486799373e173110f5c5): perf score=0.033374
I20240502 01:55:42.290601 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling FlushMRSOp(5d40847a56704b33b771c3fd87d6dfae): perf score=0.033388
I20240502 01:55:42.297765 5050 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:42.298009 5050 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:42.299124 5050 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:55:42.300288 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:55:42.300544 22251 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
W20240502 01:55:42.302979 21791 leader_election.cc:341] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 36d358202b544e74bcc378060082df4d
W20240502 01:55:42.304780 21791 leader_election.cc:334] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:42.304916 21791 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:42.305644 5050 raft_consensus.cc:2726] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:42.341325 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: FlushMRSOp(ec67f45b2b604c129d8572c5ca6acbc4) complete. Timing: real 0.116s user 0.084s sys 0.018s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8065,"dirs.run_cpu_time_us":242,"dirs.run_wall_time_us":13171,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":835,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":31822,"threads_started":2,"wal-append.queue_time_us":23970}
I20240502 01:55:42.343519 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling UndoDeltaBlockGCOp(ec67f45b2b604c129d8572c5ca6acbc4): 4921 bytes on disk
I20240502 01:55:42.344290 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: UndoDeltaBlockGCOp(ec67f45b2b604c129d8572c5ca6acbc4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240502 01:55:42.367657 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: FlushMRSOp(ba39439749a9486799373e173110f5c5) complete. Timing: real 0.136s user 0.046s sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":29676,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":23694,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":950,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":42843,"threads_started":2,"wal-append.queue_time_us":13253}
I20240502 01:55:42.370805 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling UndoDeltaBlockGCOp(ba39439749a9486799373e173110f5c5): 4581 bytes on disk
I20240502 01:55:42.375389 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: UndoDeltaBlockGCOp(ba39439749a9486799373e173110f5c5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240502 01:55:42.380486 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling FlushMRSOp(5d40847a56704b33b771c3fd87d6dfae): perf score=0.033409
I20240502 01:55:42.384546 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: FlushMRSOp(5d40847a56704b33b771c3fd87d6dfae) complete. Timing: real 0.093s user 0.049s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":10495,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":6686,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1128,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":9041,"threads_started":2,"wal-append.queue_time_us":3915}
I20240502 01:55:42.387288 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling UndoDeltaBlockGCOp(5d40847a56704b33b771c3fd87d6dfae): 4834 bytes on disk
I20240502 01:55:42.388013 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: UndoDeltaBlockGCOp(5d40847a56704b33b771c3fd87d6dfae) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240502 01:55:42.413702 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling FlushMRSOp(ec67f45b2b604c129d8572c5ca6acbc4): perf score=0.033417
W20240502 01:55:42.437636 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:42.493520 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: FlushMRSOp(ec67f45b2b604c129d8572c5ca6acbc4) complete. Timing: real 0.079s user 0.057s sys 0.020s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1994,"drs_written":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1368,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6492,"threads_started":1,"wal-append.queue_time_us":6343}
I20240502 01:55:42.496556 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling UndoDeltaBlockGCOp(ec67f45b2b604c129d8572c5ca6acbc4): 4921 bytes on disk
I20240502 01:55:42.497360 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: UndoDeltaBlockGCOp(ec67f45b2b604c129d8572c5ca6acbc4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240502 01:55:42.508325 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: FlushMRSOp(5d40847a56704b33b771c3fd87d6dfae) complete. Timing: real 0.127s user 0.052s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1347,"drs_written":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1334,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":8120,"threads_started":1,"wal-append.queue_time_us":8201}
I20240502 01:55:42.510854 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling UndoDeltaBlockGCOp(5d40847a56704b33b771c3fd87d6dfae): 4834 bytes on disk
I20240502 01:55:42.517253 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: UndoDeltaBlockGCOp(5d40847a56704b33b771c3fd87d6dfae) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":199,"lbm_reads_lt_1ms":4}
W20240502 01:55:42.543901 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:42.656505 24669 consensus_peers.cc:485] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:55:42.697144 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:42.698515 24583 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:42.712522 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:42.716236 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:42.716476 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:42.724718 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:55:42.753053 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:42.763355 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:42.767761 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:42.816627 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:42.822881 24715 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 01:55:42.838768 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:42.856168 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:42.860836 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:42.868394 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:42.898842 24669 consensus_peers.cc:485] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:42.907297 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:42.999075 24669 consensus_peers.cc:485] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:55:43.006538 24603 consensus_queue.cc:237] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:43.008529 24735 raft_consensus.cc:1261] T 2036e577d94242728ae4d00fe845121b P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:43.008553 22120 raft_consensus.cc:1261] T 2036e577d94242728ae4d00fe845121b P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:43.009166 4784 consensus_queue.cc:1035] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:43.009456 5139 consensus_queue.cc:1035] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:43.012414 4784 raft_consensus.cc:2932] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:43.013079 24735 raft_consensus.cc:2932] T 2036e577d94242728ae4d00fe845121b P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:43.016132 22120 raft_consensus.cc:2932] T 2036e577d94242728ae4d00fe845121b P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:43.021016 21711 catalog_manager.cc:5484] T 2036e577d94242728ae4d00fe845121b P 28157bc3e678428fb5f93c077d528513 reported cstate change: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
W20240502 01:55:43.030159 24669 consensus_peers.cc:485] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:43.030879 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:55:43.042600 21837 tablet_service.cc:1514] Processing DeleteTablet for tablet 2036e577d94242728ae4d00fe845121b with delete_type TABLET_DATA_TOMBSTONED (TS 759257ea7bef415ca698e60d82199d77 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47656
I20240502 01:55:43.044553 5168 tablet_replica.cc:330] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:55:43.045250 5168 raft_consensus.cc:2229] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:43.045565 5168 raft_consensus.cc:2258] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:43.046339 5168 ts_tablet_manager.cc:1875] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:43.053367 5168 ts_tablet_manager.cc:1888] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:43.053508 5168 log.cc:1196] T 2036e577d94242728ae4d00fe845121b P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/2036e577d94242728ae4d00fe845121b
I20240502 01:55:43.054426 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet 2036e577d94242728ae4d00fe845121b (table rebalancer_extra_table_42 [id=64a3018c59994b83b4283de8ca4e1f27]) successfully deleted
W20240502 01:55:43.265784 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:43.338948 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:43.348196 24715 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:43.356405 24583 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:43.358237 24669 consensus_peers.cc:485] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:55:43.400172 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:43.412441 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:43.477046 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:43.480247 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:43.775436 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:43.831058 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:43.892529 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:43.904134 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:43.910974 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:43.957978 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:43.964455 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:43.981573 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:55:43.989483 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:44.025833 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:44.026940 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:44.027052 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:44.031335 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:55:44.032495 24583 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:44.057178 22185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 107 similar messages]
I20240502 01:55:44.060354 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 01:55:44.063155 24537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 01:55:44.065727 21923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 01:55:44.066843 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:55:44.095332 24715 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:44.097718 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:44.106932 24669 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 01:55:44.108036 24669 consensus_peers.cc:485] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:44.122481 24934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240502 01:55:44.123873 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:44.130553 22054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240502 01:55:44.165611 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 01:55:44.166147 21791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20240502 01:55:44.167299 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:44.167760 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:44.175335 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:44.219642 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:44.219786 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:55:44.221297 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:44.224332 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling FlushMRSOp(4b924192d8da4a8bb7df2e6c5e4cbd4f): perf score=0.033336
W20240502 01:55:44.231585 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:44.245306 24669 consensus_peers.cc:485] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:44.260120 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:55:44.267273 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling FlushMRSOp(166e717fb9624ae58adb6699790daf7a): perf score=0.033360
I20240502 01:55:44.287166 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling FlushMRSOp(4b924192d8da4a8bb7df2e6c5e4cbd4f): perf score=0.033361
W20240502 01:55:44.318852 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:44.326287 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:44.331815 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: FlushMRSOp(4b924192d8da4a8bb7df2e6c5e4cbd4f) complete. Timing: real 0.107s user 0.049s sys 0.030s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":25089,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1915,"drs_written":1,"lbm_read_time_us":155,"lbm_reads_lt_1ms":4,"lbm_write_time_us":973,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":31453,"threads_started":2,"wal-append.queue_time_us":6330}
I20240502 01:55:44.334914 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling UndoDeltaBlockGCOp(4b924192d8da4a8bb7df2e6c5e4cbd4f): 4921 bytes on disk
W20240502 01:55:44.342157 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:55:44.351117 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: UndoDeltaBlockGCOp(4b924192d8da4a8bb7df2e6c5e4cbd4f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":149,"lbm_reads_lt_1ms":4}
W20240502 01:55:44.355135 24669 consensus_peers.cc:485] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:44.372589 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:55:44.382234 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling FlushMRSOp(72f46ec6ed7848438cb9ad6e38053c4d): perf score=0.033382
I20240502 01:55:44.389496 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: FlushMRSOp(166e717fb9624ae58adb6699790daf7a) complete. Timing: real 0.122s user 0.044s sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":11815,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":8138,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":942,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":21512,"threads_started":2,"wal-append.queue_time_us":9699}
I20240502 01:55:44.392565 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling FlushMRSOp(72f46ec6ed7848438cb9ad6e38053c4d): perf score=0.033388
W20240502 01:55:44.406019 24669 consensus_peers.cc:485] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:55:44.430624 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: FlushMRSOp(4b924192d8da4a8bb7df2e6c5e4cbd4f) complete. Timing: real 0.141s user 0.038s sys 0.014s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":17999,"dirs.run_cpu_time_us":255,"dirs.run_wall_time_us":12693,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":958,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":25965,"threads_started":2,"wal-append.queue_time_us":7628}
I20240502 01:55:44.434132 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling UndoDeltaBlockGCOp(4b924192d8da4a8bb7df2e6c5e4cbd4f): 4921 bytes on disk
I20240502 01:55:44.434803 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: UndoDeltaBlockGCOp(4b924192d8da4a8bb7df2e6c5e4cbd4f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4}
I20240502 01:55:44.438741 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling FlushMRSOp(72f46ec6ed7848438cb9ad6e38053c4d): perf score=0.033399
W20240502 01:55:44.480612 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:44.481822 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:55:44.494436 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: FlushMRSOp(72f46ec6ed7848438cb9ad6e38053c4d) complete. Timing: real 0.108s user 0.073s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":12116,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1267,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":905,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":18627,"threads_started":2,"wal-append.queue_time_us":6479}
I20240502 01:55:44.497083 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling UndoDeltaBlockGCOp(72f46ec6ed7848438cb9ad6e38053c4d): 4834 bytes on disk
I20240502 01:55:44.498113 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: UndoDeltaBlockGCOp(72f46ec6ed7848438cb9ad6e38053c4d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240502 01:55:44.498091 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: FlushMRSOp(72f46ec6ed7848438cb9ad6e38053c4d) complete. Timing: real 0.059s user 0.049s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1107,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":939,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6515,"threads_started":1,"wal-append.queue_time_us":6358}
I20240502 01:55:44.500516 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling UndoDeltaBlockGCOp(72f46ec6ed7848438cb9ad6e38053c4d): 4834 bytes on disk
W20240502 01:55:44.501605 24669 consensus_peers.cc:485] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:55:44.502135 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: UndoDeltaBlockGCOp(72f46ec6ed7848438cb9ad6e38053c4d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240502 01:55:44.507838 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling FlushMRSOp(166e717fb9624ae58adb6699790daf7a): perf score=0.033415
W20240502 01:55:44.524591 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:44.535997 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:55:44.568998 5140 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:44.569237 5140 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:44.570132 5140 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
W20240502 01:55:44.572283 21791 leader_election.cc:341] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 36d358202b544e74bcc378060082df4d
I20240502 01:55:44.572932 22248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:55:44.573258 22248 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
W20240502 01:55:44.574782 21791 leader_election.cc:334] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:44.574910 21791 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:44.578171 4888 raft_consensus.cc:2726] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:44.589529 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: FlushMRSOp(72f46ec6ed7848438cb9ad6e38053c4d) complete. Timing: real 0.197s user 0.088s sys 0.052s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":356,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":3697,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1706,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":72113,"threads_started":1,"wal-append.queue_time_us":72219}
I20240502 01:55:44.598018 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling UndoDeltaBlockGCOp(72f46ec6ed7848438cb9ad6e38053c4d): 4834 bytes on disk
I20240502 01:55:44.598745 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: UndoDeltaBlockGCOp(72f46ec6ed7848438cb9ad6e38053c4d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
I20240502 01:55:44.601523 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling UndoDeltaBlockGCOp(166e717fb9624ae58adb6699790daf7a): 4581 bytes on disk
I20240502 01:55:44.602305 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: UndoDeltaBlockGCOp(166e717fb9624ae58adb6699790daf7a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":138,"lbm_reads_lt_1ms":4}
I20240502 01:55:44.619304 22248 consensus_queue.cc:237] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:44.621212 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: FlushMRSOp(166e717fb9624ae58adb6699790daf7a) complete. Timing: real 0.113s user 0.045s sys 0.026s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1382,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1137,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":332,"threads_started":1,"wal-append.queue_time_us":439}
I20240502 01:55:44.622292 22118 raft_consensus.cc:1261] T ad8bc5be633d4646a7c10d2483ed4e5b P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:44.622299 21985 raft_consensus.cc:1261] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:44.622972 4647 consensus_queue.cc:1035] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:44.623808 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling UndoDeltaBlockGCOp(166e717fb9624ae58adb6699790daf7a): 4581 bytes on disk
I20240502 01:55:44.624402 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: UndoDeltaBlockGCOp(166e717fb9624ae58adb6699790daf7a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240502 01:55:44.624750 4647 consensus_queue.cc:1035] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:44.626224 5288 raft_consensus.cc:2932] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:44.630400 21985 raft_consensus.cc:2932] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:44.633685 22118 raft_consensus.cc:2932] T ad8bc5be633d4646a7c10d2483ed4e5b P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:44.637295 21712 catalog_manager.cc:5484] T ad8bc5be633d4646a7c10d2483ed4e5b P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index 2 to 3, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
W20240502 01:55:44.641007 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:55:44.649982 24981 tablet_service.cc:1514] Processing DeleteTablet for tablet ad8bc5be633d4646a7c10d2483ed4e5b with delete_type TABLET_DATA_TOMBSTONED (TS 41c70f373c9d49a49891b03a12c6cb2b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:39964
I20240502 01:55:44.650585 5344 tablet_replica.cc:330] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:55:44.650892 5344 raft_consensus.cc:2229] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:44.651192 5344 raft_consensus.cc:2258] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:44.651866 5344 ts_tablet_manager.cc:1875] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 01:55:44.661161 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:55:44.662436 5344 ts_tablet_manager.cc:1888] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:44.662699 5344 log.cc:1196] T ad8bc5be633d4646a7c10d2483ed4e5b P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/ad8bc5be633d4646a7c10d2483ed4e5b
I20240502 01:55:44.663587 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet ad8bc5be633d4646a7c10d2483ed4e5b (table rebalancer_extra_table_30 [id=f03cad8416b24f2c8dc87cdc63843ec7]) successfully deleted
W20240502 01:55:44.664983 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:44.731333 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:44.831288 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:44.885718 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:55:44.932209 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:44.989637 24583 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:44.996258 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:45.088922 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:45.131665 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:45.131772 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:55:45.133224 24715 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:45.143517 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 01:55:45.162534 24669 consensus_peers.cc:485] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:55:45.178655 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240502 01:55:45.242502 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:45.248608 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:45.268185 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:55:45.294533 24999 consensus_queue.cc:237] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:45.296862 24733 raft_consensus.cc:1261] T f018db81d6214bd0ac12feaf54541fed P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:45.296983 21988 raft_consensus.cc:1261] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:45.297405 4246 consensus_queue.cc:1035] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:45.297659 4921 consensus_queue.cc:1035] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:45.300688 4246 raft_consensus.cc:2932] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
W20240502 01:55:45.301579 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:45.301393 21988 raft_consensus.cc:2932] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:45.302837 24733 raft_consensus.cc:2932] T f018db81d6214bd0ac12feaf54541fed P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:45.310129 21712 catalog_manager.cc:5484] T f018db81d6214bd0ac12feaf54541fed P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index 2 to 3, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) evicted. New cstate: current_term: 1 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:45.334239 21837 tablet_service.cc:1514] Processing DeleteTablet for tablet f018db81d6214bd0ac12feaf54541fed with delete_type TABLET_DATA_TOMBSTONED (TS 759257ea7bef415ca698e60d82199d77 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47656
I20240502 01:55:45.369606 5430 tablet_replica.cc:330] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77: stopping tablet replica
I20240502 01:55:45.371949 5430 raft_consensus.cc:2229] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:45.372344 5430 raft_consensus.cc:2258] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:45.373066 5430 ts_tablet_manager.cc:1875] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:45.381891 5430 ts_tablet_manager.cc:1888] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:45.382117 5430 log.cc:1196] T f018db81d6214bd0ac12feaf54541fed P 759257ea7bef415ca698e60d82199d77: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/wals/f018db81d6214bd0ac12feaf54541fed
I20240502 01:55:45.383064 21699 catalog_manager.cc:4830] TS 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): tablet f018db81d6214bd0ac12feaf54541fed (table rebalancer_extra_table_38 [id=b6620e4ad5274690b79a7b8e7e0523bb]) successfully deleted
W20240502 01:55:45.448758 24669 consensus_peers.cc:485] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:45.470964 24669 consensus_peers.cc:485] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:45.477699 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:45.495728 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:45.497949 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:45.513507 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:45.530190 24669 consensus_peers.cc:485] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:45.750463 24669 consensus_peers.cc:485] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:55:45.758049 24583 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:45.762771 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:45.812541 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:45.815584 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:45.834775 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:45.834959 24715 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 01:55:45.856168 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:45.873973 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:46.199658 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.277087 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.298202 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:55:46.299139 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:46.344314 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 01:55:46.350617 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.370596 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:46.372082 24715 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:46.385277 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 01:55:46.391088 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.403029 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.403314 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.422755 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.469415 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:55:46.469902 24583 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:46.472992 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:46.474845 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:55:46.530622 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240502 01:55:46.536592 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.576850 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:55:46.607026 5339 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:46.607204 5339 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:46.608206 5339 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
W20240502 01:55:46.609860 22185 leader_election.cc:334] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:46.610018 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:55:46.610255 21854 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
W20240502 01:55:46.612200 22185 leader_election.cc:341] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 36d358202b544e74bcc378060082df4d
I20240502 01:55:46.612324 22185 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:46.612943 5432 raft_consensus.cc:2726] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:55:46.636623 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.643697 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.685384 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.708860 24669 consensus_peers.cc:485] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.768245 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.774780 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.804164 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.819545 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.853832 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:46.898141 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:46.902678 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:55:46.909089 24715 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 01:55:46.926570 24669 consensus_peers.cc:485] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:55:46.928017 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:46.930210 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:46.942936 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 01:55:46.943321 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.945570 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.957250 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.959328 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:46.960484 24669 consensus_peers.cc:485] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:55:46.967171 24583 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:46.969143 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:46.981518 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:47.049786 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:47.139798 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:47.139873 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:47.156957 24669 consensus_peers.cc:485] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:47.162043 24669 consensus_peers.cc:485] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:55:47.166883 22118 consensus_queue.cc:237] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:47.169437 24603 raft_consensus.cc:1261] T 88457f21ad4a4e00bb372479c13963ec P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:47.169823 21985 raft_consensus.cc:1261] T 88457f21ad4a4e00bb372479c13963ec P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:47.171869 5608 consensus_queue.cc:1035] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:47.174518 5608 consensus_queue.cc:1035] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:47.176542 4767 raft_consensus.cc:2932] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:47.177266 24603 raft_consensus.cc:2932] T 88457f21ad4a4e00bb372479c13963ec P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:47.183413 21710 catalog_manager.cc:5484] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New cstate: current_term: 1 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:47.186081 21988 raft_consensus.cc:2932] T 88457f21ad4a4e00bb372479c13963ec P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:47.200489 22229 tablet_service.cc:1514] Processing DeleteTablet for tablet 88457f21ad4a4e00bb372479c13963ec with delete_type TABLET_DATA_TOMBSTONED (TS c2a28baa7ee2469294d94b7fe5455cff not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50898
I20240502 01:55:47.201279 5656 tablet_replica.cc:330] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff: stopping tablet replica
I20240502 01:55:47.201493 5656 raft_consensus.cc:2229] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:47.201740 5656 raft_consensus.cc:2258] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:47.202824 5656 ts_tablet_manager.cc:1875] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:47.219360 5656 ts_tablet_manager.cc:1888] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:55:47.219468 5656 log.cc:1196] T 88457f21ad4a4e00bb372479c13963ec P c2a28baa7ee2469294d94b7fe5455cff: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/wals/88457f21ad4a4e00bb372479c13963ec
I20240502 01:55:47.220160 21701 catalog_manager.cc:4830] TS c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): tablet 88457f21ad4a4e00bb372479c13963ec (table rebalance_test_table_3 [id=27064ffcfcc643d6ad082b6cd6ea21db]) successfully deleted
W20240502 01:55:47.255344 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:47.315243 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:47.337200 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:47.454301 24669 consensus_peers.cc:485] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:47.511507 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:47.527619 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:55:47.533427 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling FlushMRSOp(8a0ae0d8e8784db9988b3469d318f7fb): perf score=0.033366
I20240502 01:55:47.542171 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:47.580688 24583 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:47.581998 24715 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 01:55:47.587523 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:55:47.597833 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:47.639797 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling FlushMRSOp(3964f5f080f145a5923dac2f076637c8): perf score=0.033392
I20240502 01:55:47.648001 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling FlushMRSOp(8a0ae0d8e8784db9988b3469d318f7fb): perf score=0.033398
W20240502 01:55:47.655258 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:55:47.668802 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling FlushMRSOp(9bceab60366b4ade98f86072d391e7f4): perf score=0.033396
W20240502 01:55:47.672354 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:47.676476 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:55:47.690428 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:47.709126 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:47.729594 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:47.732012 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:47.777350 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:47.833336 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: FlushMRSOp(8a0ae0d8e8784db9988b3469d318f7fb) complete. Timing: real 0.300s user 0.085s sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":108245,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":4986,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1013,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":132028,"threads_started":2,"wal-append.queue_time_us":22389}
I20240502 01:55:47.856304 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: FlushMRSOp(9bceab60366b4ade98f86072d391e7f4) complete. Timing: real 0.187s user 0.060s sys 0.022s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6589,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":13355,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4722,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":13241,"threads_started":2,"wal-append.queue_time_us":6401}
I20240502 01:55:47.856534 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling FlushMRSOp(3964f5f080f145a5923dac2f076637c8): perf score=0.033452
I20240502 01:55:47.859102 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling UndoDeltaBlockGCOp(9bceab60366b4ade98f86072d391e7f4): 4834 bytes on disk
I20240502 01:55:47.861411 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: FlushMRSOp(8a0ae0d8e8784db9988b3469d318f7fb) complete. Timing: real 0.213s user 0.059s sys 0.026s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6340,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":5746,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1018,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":13760,"threads_started":2,"wal-append.queue_time_us":7195}
I20240502 01:55:47.864099 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling UndoDeltaBlockGCOp(8a0ae0d8e8784db9988b3469d318f7fb): 4921 bytes on disk
I20240502 01:55:47.865135 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: UndoDeltaBlockGCOp(8a0ae0d8e8784db9988b3469d318f7fb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":295,"lbm_reads_lt_1ms":4}
I20240502 01:55:47.869807 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: UndoDeltaBlockGCOp(9bceab60366b4ade98f86072d391e7f4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
W20240502 01:55:47.913544 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:47.915642 24669 consensus_peers.cc:485] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:55:47.967918 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: FlushMRSOp(3964f5f080f145a5923dac2f076637c8) complete. Timing: real 0.328s user 0.055s sys 0.028s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":57939,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":8730,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1264,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":73213,"threads_started":2,"wal-append.queue_time_us":10571}
I20240502 01:55:47.970978 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling UndoDeltaBlockGCOp(3964f5f080f145a5923dac2f076637c8): 4581 bytes on disk
I20240502 01:55:47.971801 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: UndoDeltaBlockGCOp(3964f5f080f145a5923dac2f076637c8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240502 01:55:47.993417 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: FlushMRSOp(3964f5f080f145a5923dac2f076637c8) complete. Timing: real 0.136s user 0.038s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":257,"dirs.run_wall_time_us":1550,"drs_written":1,"lbm_read_time_us":137,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9181,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":4694,"threads_started":1,"wal-append.queue_time_us":4828}
I20240502 01:55:47.999692 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling UndoDeltaBlockGCOp(8a0ae0d8e8784db9988b3469d318f7fb): 4921 bytes on disk
I20240502 01:55:48.000481 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: UndoDeltaBlockGCOp(8a0ae0d8e8784db9988b3469d318f7fb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240502 01:55:48.002606 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling UndoDeltaBlockGCOp(3964f5f080f145a5923dac2f076637c8): 4581 bytes on disk
I20240502 01:55:48.003218 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: UndoDeltaBlockGCOp(3964f5f080f145a5923dac2f076637c8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
W20240502 01:55:48.012732 24669 consensus_peers.cc:485] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:55:48.042955 21854 consensus_queue.cc:237] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
W20240502 01:55:48.044735 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:55:48.045641 24603 raft_consensus.cc:1261] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:48.045617 21988 raft_consensus.cc:1261] T 72f46ec6ed7848438cb9ad6e38053c4d P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:48.046342 5140 consensus_queue.cc:1035] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:48.046651 5624 consensus_queue.cc:1035] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:48.049636 5624 raft_consensus.cc:2932] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:48.050420 24601 raft_consensus.cc:2932] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:48.053872 21988 raft_consensus.cc:2932] T 72f46ec6ed7848438cb9ad6e38053c4d P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:48.056619 21706 catalog_manager.cc:5484] T 72f46ec6ed7848438cb9ad6e38053c4d P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
W20240502 01:55:48.061568 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:55:48.069341 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 72f46ec6ed7848438cb9ad6e38053c4d with delete_type TABLET_DATA_TOMBSTONED (TS c64840657cdf4b93980bd34120c94d9a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40366
I20240502 01:55:48.070153 5770 tablet_replica.cc:330] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:55:48.070472 5770 raft_consensus.cc:2229] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:48.070775 5770 raft_consensus.cc:2258] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:48.071854 5770 ts_tablet_manager.cc:1875] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:48.080637 5770 ts_tablet_manager.cc:1888] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:55:48.080827 5770 log.cc:1196] T 72f46ec6ed7848438cb9ad6e38053c4d P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/72f46ec6ed7848438cb9ad6e38053c4d
I20240502 01:55:48.081694 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet 72f46ec6ed7848438cb9ad6e38053c4d (table rebalance_test_table_9 [id=714c5d94b5b64cb29dc398713b1727fe]) successfully deleted
W20240502 01:55:48.097678 24669 consensus_peers.cc:485] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:55:48.145884 21854 consensus_queue.cc:237] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
W20240502 01:55:48.147511 24669 consensus_peers.cc:485] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:55:48.149204 25001 raft_consensus.cc:1261] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:48.149240 21985 raft_consensus.cc:1261] T ae04b73f8a574264a26a647400a44235 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:55:48.149523 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae04b73f8a574264a26a647400a44235. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:48.149863 5624 consensus_queue.cc:1035] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:48.150432 5624 consensus_queue.cc:1035] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:55:48.165877 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:48.166299 5140 raft_consensus.cc:2932] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:48.167244 21985 raft_consensus.cc:2932] T ae04b73f8a574264a26a647400a44235 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:48.173050 25001 raft_consensus.cc:2932] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:48.175576 21713 catalog_manager.cc:5484] T ae04b73f8a574264a26a647400a44235 P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:48.244613 24601 consensus_queue.cc:237] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:48.246841 24999 raft_consensus.cc:1261] T 2362b0bec07540f1bc0efedbcf61599c P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:55:48.247433 24536 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2362b0bec07540f1bc0efedbcf61599c. This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:48.247612 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:48.247663 5563 consensus_queue.cc:1035] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:55:48.348491 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:48.451840 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:55:48.476969 5733 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:48.477202 5733 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:48.478015 5733 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:55:48.478626 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:55:48.479023 21854 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
W20240502 01:55:48.479239 22185 leader_election.cc:341] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 36d358202b544e74bcc378060082df4d
W20240502 01:55:48.480923 22185 leader_election.cc:334] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:48.481084 22185 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:48.481369 5733 raft_consensus.cc:2726] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:48.531473 5821 ts_tablet_manager.cc:902] T ae04b73f8a574264a26a647400a44235 P 28157bc3e678428fb5f93c077d528513: Initiating tablet copy from peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:55:48.541427 5821 tablet_copy_client.cc:308] T ae04b73f8a574264a26a647400a44235 P 28157bc3e678428fb5f93c077d528513: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.193:33785
I20240502 01:55:48.542236 21878 tablet_copy_service.cc:140] P 759257ea7bef415ca698e60d82199d77: Received BeginTabletCopySession request for tablet ae04b73f8a574264a26a647400a44235 from peer 28157bc3e678428fb5f93c077d528513 ({username='slave'} at 127.14.212.198:60613)
I20240502 01:55:48.542433 21878 tablet_copy_service.cc:161] P 759257ea7bef415ca698e60d82199d77: Beginning new tablet copy session on tablet ae04b73f8a574264a26a647400a44235 from peer 28157bc3e678428fb5f93c077d528513 at {username='slave'} at 127.14.212.198:60613: session id = 28157bc3e678428fb5f93c077d528513-ae04b73f8a574264a26a647400a44235
I20240502 01:55:48.545229 21878 tablet_copy_source_session.cc:205] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:48.546881 5821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae04b73f8a574264a26a647400a44235. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:48.553678 5821 tablet_copy_client.cc:783] T ae04b73f8a574264a26a647400a44235 P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:48.554087 5821 tablet_copy_client.cc:647] T ae04b73f8a574264a26a647400a44235 P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:48.558193 5821 tablet_copy_client.cc:523] T ae04b73f8a574264a26a647400a44235 P 28157bc3e678428fb5f93c077d528513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:48.561072 5821 tablet_bootstrap.cc:492] T ae04b73f8a574264a26a647400a44235 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:55:48.565862 5821 tablet_bootstrap.cc:492] T ae04b73f8a574264a26a647400a44235 P 28157bc3e678428fb5f93c077d528513: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:48.566241 5821 tablet_bootstrap.cc:492] T ae04b73f8a574264a26a647400a44235 P 28157bc3e678428fb5f93c077d528513: Bootstrap complete.
I20240502 01:55:48.566459 5821 ts_tablet_manager.cc:1367] T ae04b73f8a574264a26a647400a44235 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 01:55:48.567101 5821 raft_consensus.cc:348] T ae04b73f8a574264a26a647400a44235 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:48.567412 5821 raft_consensus.cc:729] T ae04b73f8a574264a26a647400a44235 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:55:48.567696 5821 consensus_queue.cc:260] T ae04b73f8a574264a26a647400a44235 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:48.568609 5821 ts_tablet_manager.cc:1398] T ae04b73f8a574264a26a647400a44235 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:55:48.570995 21878 tablet_copy_service.cc:342] P 759257ea7bef415ca698e60d82199d77: Request end of tablet copy session 28157bc3e678428fb5f93c077d528513-ae04b73f8a574264a26a647400a44235 received from {username='slave'} at 127.14.212.198:60613
I20240502 01:55:48.571095 21878 tablet_copy_service.cc:433] P 759257ea7bef415ca698e60d82199d77: ending tablet copy session 28157bc3e678428fb5f93c077d528513-ae04b73f8a574264a26a647400a44235 on tablet ae04b73f8a574264a26a647400a44235 with peer 28157bc3e678428fb5f93c077d528513
W20240502 01:55:48.693797 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:48.767093 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:48.772729 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:55:48.795058 5830 ts_tablet_manager.cc:902] T 2362b0bec07540f1bc0efedbcf61599c P 759257ea7bef415ca698e60d82199d77: Initiating tablet copy from peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:55:48.796177 5830 tablet_copy_client.cc:308] T 2362b0bec07540f1bc0efedbcf61599c P 759257ea7bef415ca698e60d82199d77: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.197:36677
I20240502 01:55:48.796941 24623 tablet_copy_service.cc:140] P 0c64d40b5d7644bd8b29feaa80eab72f: Received BeginTabletCopySession request for tablet 2362b0bec07540f1bc0efedbcf61599c from peer 759257ea7bef415ca698e60d82199d77 ({username='slave'} at 127.14.212.193:46525)
I20240502 01:55:48.797112 24623 tablet_copy_service.cc:161] P 0c64d40b5d7644bd8b29feaa80eab72f: Beginning new tablet copy session on tablet 2362b0bec07540f1bc0efedbcf61599c from peer 759257ea7bef415ca698e60d82199d77 at {username='slave'} at 127.14.212.193:46525: session id = 759257ea7bef415ca698e60d82199d77-2362b0bec07540f1bc0efedbcf61599c
I20240502 01:55:48.799695 24623 tablet_copy_source_session.cc:205] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:48.801508 5830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2362b0bec07540f1bc0efedbcf61599c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:48.806990 5830 tablet_copy_client.cc:783] T 2362b0bec07540f1bc0efedbcf61599c P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:48.807236 5830 tablet_copy_client.cc:647] T 2362b0bec07540f1bc0efedbcf61599c P 759257ea7bef415ca698e60d82199d77: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:48.809011 5830 tablet_copy_client.cc:523] T 2362b0bec07540f1bc0efedbcf61599c P 759257ea7bef415ca698e60d82199d77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 01:55:48.812196 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:55:48.813118 5830 tablet_bootstrap.cc:492] T 2362b0bec07540f1bc0efedbcf61599c P 759257ea7bef415ca698e60d82199d77: Bootstrap starting.
I20240502 01:55:48.818089 5830 tablet_bootstrap.cc:492] T 2362b0bec07540f1bc0efedbcf61599c P 759257ea7bef415ca698e60d82199d77: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:55:48.818436 5830 tablet_bootstrap.cc:492] T 2362b0bec07540f1bc0efedbcf61599c P 759257ea7bef415ca698e60d82199d77: Bootstrap complete.
I20240502 01:55:48.818660 5830 ts_tablet_manager.cc:1367] T 2362b0bec07540f1bc0efedbcf61599c P 759257ea7bef415ca698e60d82199d77: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:55:48.819231 5830 raft_consensus.cc:348] T 2362b0bec07540f1bc0efedbcf61599c P 759257ea7bef415ca698e60d82199d77 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:48.819973 5830 raft_consensus.cc:729] T 2362b0bec07540f1bc0efedbcf61599c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759257ea7bef415ca698e60d82199d77, State: Initialized, Role: FOLLOWER
I20240502 01:55:48.820185 5830 consensus_queue.cc:260] T 2362b0bec07540f1bc0efedbcf61599c P 759257ea7bef415ca698e60d82199d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:48.820758 5830 ts_tablet_manager.cc:1398] T 2362b0bec07540f1bc0efedbcf61599c P 759257ea7bef415ca698e60d82199d77: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 01:55:48.821555 24623 tablet_copy_service.cc:342] P 0c64d40b5d7644bd8b29feaa80eab72f: Request end of tablet copy session 759257ea7bef415ca698e60d82199d77-2362b0bec07540f1bc0efedbcf61599c received from {username='slave'} at 127.14.212.193:46525
I20240502 01:55:48.821720 24623 tablet_copy_service.cc:433] P 0c64d40b5d7644bd8b29feaa80eab72f: ending tablet copy session 759257ea7bef415ca698e60d82199d77-2362b0bec07540f1bc0efedbcf61599c on tablet 2362b0bec07540f1bc0efedbcf61599c with peer 759257ea7bef415ca698e60d82199d77
W20240502 01:55:48.916307 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:48.939006 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:48.945261 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:48.969331 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:48.977890 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.003790 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.005373 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.009215 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.062758 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:55:49.067046 24735 raft_consensus.cc:1203] T ae04b73f8a574264a26a647400a44235 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 01:55:49.093777 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.099038 24669 consensus_peers.cc:485] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.120003 22185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 01:55:49.120285 24669 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240502 01:55:49.138566 24537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 01:55:49.152523 21923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 01:55:49.199888 21854 raft_consensus.cc:1203] T 2362b0bec07540f1bc0efedbcf61599c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:49.202236 5837 raft_consensus.cc:2932] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:49.203274 21854 raft_consensus.cc:2932] T 2362b0bec07540f1bc0efedbcf61599c P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:49.212863 21711 catalog_manager.cc:5484] T 2362b0bec07540f1bc0efedbcf61599c P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New cstate: current_term: 1 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:49.215081 24999 raft_consensus.cc:2932] T 2362b0bec07540f1bc0efedbcf61599c P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 759257ea7bef415ca698e60d82199d77 (127.14.212.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
W20240502 01:55:49.244334 21791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 01:55:49.245324 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:49.245534 22054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 01:55:49.251428 24669 consensus_peers.cc:485] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:55:49.254120 22248 consensus_queue.cc:237] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:49.256923 22120 raft_consensus.cc:1261] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:49.256938 21988 raft_consensus.cc:1261] T de3eb3ed981d4f629eaa7f473ea3b430 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:49.257186 22185 consensus_peers.cc:485] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Couldn't send request to peer 0c64d40b5d7644bd8b29feaa80eab72f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de3eb3ed981d4f629eaa7f473ea3b430. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:49.257536 5644 consensus_queue.cc:1035] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:49.257927 5829 consensus_queue.cc:1035] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:49.265329 5644 raft_consensus.cc:2932] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:49.269855 22120 raft_consensus.cc:2932] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:49.272553 21988 raft_consensus.cc:2932] T de3eb3ed981d4f629eaa7f473ea3b430 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
W20240502 01:55:49.274890 24934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 01:55:49.280645 21706 catalog_manager.cc:5484] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a reported cstate change: config changed from index -1 to 4, VOTER 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197) added. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
W20240502 01:55:49.285916 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.296542 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.300117 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.347931 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.352144 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:55:49.356746 21988 consensus_queue.cc:237] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:49.359786 24601 raft_consensus.cc:1261] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:55:49.360456 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bddb4a31d81d44afaabc247fd1449767. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:49.361096 5306 consensus_queue.cc:1035] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:55:49.361567 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:49.454299 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.459889 24669 consensus_peers.cc:485] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.506330 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.575798 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.582509 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.595932 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.618779 24669 consensus_peers.cc:485] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.621351 24669 consensus_peers.cc:485] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.623270 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.660946 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:49.675451 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:55:49.741168 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling FlushMRSOp(3be66f4b90824a21a5f41915c9041970): perf score=0.033344
W20240502 01:55:49.753121 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:55:49.770960 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling FlushMRSOp(fedbec8860904a2cb5294037b7a4bcad): perf score=0.033352
I20240502 01:55:49.782321 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: FlushMRSOp(3be66f4b90824a21a5f41915c9041970) complete. Timing: real 0.041s user 0.034s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":364,"dirs.run_cpu_time_us":273,"dirs.run_wall_time_us":1578,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":824,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1155,"threads_started":2,"wal-append.queue_time_us":549}
I20240502 01:55:49.784907 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling UndoDeltaBlockGCOp(3be66f4b90824a21a5f41915c9041970): 4581 bytes on disk
I20240502 01:55:49.785821 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: UndoDeltaBlockGCOp(3be66f4b90824a21a5f41915c9041970) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4}
I20240502 01:55:49.788316 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling FlushMRSOp(d81f9cab671a4258aee1dd4ea4a5ee6b): perf score=0.033351
W20240502 01:55:49.835964 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:55:49.837298 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: FlushMRSOp(d81f9cab671a4258aee1dd4ea4a5ee6b) complete. Timing: real 0.049s user 0.048s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":72,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1411,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":870,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":244,"threads_started":1,"wal-append.queue_time_us":368}
I20240502 01:55:49.839638 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling UndoDeltaBlockGCOp(d81f9cab671a4258aee1dd4ea4a5ee6b): 4919 bytes on disk
I20240502 01:55:49.840399 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: UndoDeltaBlockGCOp(d81f9cab671a4258aee1dd4ea4a5ee6b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
I20240502 01:55:49.850668 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: FlushMRSOp(fedbec8860904a2cb5294037b7a4bcad) complete. Timing: real 0.079s user 0.051s sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":4004,"dirs.run_cpu_time_us":233,"dirs.run_wall_time_us":1346,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1424,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":4195,"threads_started":2,"wal-append.queue_time_us":426}
I20240502 01:55:49.853193 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling UndoDeltaBlockGCOp(fedbec8860904a2cb5294037b7a4bcad): 4834 bytes on disk
I20240502 01:55:49.853740 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: UndoDeltaBlockGCOp(fedbec8860904a2cb5294037b7a4bcad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240502 01:55:49.876389 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:55:49.882701 5891 ts_tablet_manager.cc:902] T de3eb3ed981d4f629eaa7f473ea3b430 P 0c64d40b5d7644bd8b29feaa80eab72f: Initiating tablet copy from peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:55:49.884171 5891 tablet_copy_client.cc:308] T de3eb3ed981d4f629eaa7f473ea3b430 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.196:43093
I20240502 01:55:49.884987 22271 tablet_copy_service.cc:140] P c2a28baa7ee2469294d94b7fe5455cff: Received BeginTabletCopySession request for tablet de3eb3ed981d4f629eaa7f473ea3b430 from peer 0c64d40b5d7644bd8b29feaa80eab72f ({username='slave'} at 127.14.212.197:36211)
I20240502 01:55:49.885179 22271 tablet_copy_service.cc:161] P c2a28baa7ee2469294d94b7fe5455cff: Beginning new tablet copy session on tablet de3eb3ed981d4f629eaa7f473ea3b430 from peer 0c64d40b5d7644bd8b29feaa80eab72f at {username='slave'} at 127.14.212.197:36211: session id = 0c64d40b5d7644bd8b29feaa80eab72f-de3eb3ed981d4f629eaa7f473ea3b430
I20240502 01:55:49.888029 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling FlushMRSOp(fedbec8860904a2cb5294037b7a4bcad): perf score=0.033388
I20240502 01:55:49.888109 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling FlushMRSOp(d81f9cab671a4258aee1dd4ea4a5ee6b): perf score=0.033390
I20240502 01:55:49.894403 22271 tablet_copy_source_session.cc:205] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:49.897956 5891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3eb3ed981d4f629eaa7f473ea3b430. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:55:49.902930 24669 consensus_peers.cc:485] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:49.907500 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:55:49.932948 5896 ts_tablet_manager.cc:902] T bddb4a31d81d44afaabc247fd1449767 P 28157bc3e678428fb5f93c077d528513: Initiating tablet copy from peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:55:49.940601 5891 tablet_copy_client.cc:783] T de3eb3ed981d4f629eaa7f473ea3b430 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:49.941282 5891 tablet_copy_client.cc:647] T de3eb3ed981d4f629eaa7f473ea3b430 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:49.955224 5896 tablet_copy_client.cc:308] T bddb4a31d81d44afaabc247fd1449767 P 28157bc3e678428fb5f93c077d528513: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:44411
I20240502 01:55:49.955562 5891 tablet_copy_client.cc:523] T de3eb3ed981d4f629eaa7f473ea3b430 P 0c64d40b5d7644bd8b29feaa80eab72f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:49.956060 22010 tablet_copy_service.cc:140] P 06543b05a16f42099bc86db2a0284412: Received BeginTabletCopySession request for tablet bddb4a31d81d44afaabc247fd1449767 from peer 28157bc3e678428fb5f93c077d528513 ({username='slave'} at 127.14.212.198:53247)
I20240502 01:55:49.956260 22010 tablet_copy_service.cc:161] P 06543b05a16f42099bc86db2a0284412: Beginning new tablet copy session on tablet bddb4a31d81d44afaabc247fd1449767 from peer 28157bc3e678428fb5f93c077d528513 at {username='slave'} at 127.14.212.198:53247: session id = 28157bc3e678428fb5f93c077d528513-bddb4a31d81d44afaabc247fd1449767
I20240502 01:55:49.962460 22010 tablet_copy_source_session.cc:205] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:49.965363 5891 tablet_bootstrap.cc:492] T de3eb3ed981d4f629eaa7f473ea3b430 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap starting.
I20240502 01:55:49.965482 5896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bddb4a31d81d44afaabc247fd1449767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:49.973598 5896 tablet_copy_client.cc:783] T bddb4a31d81d44afaabc247fd1449767 P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:49.973865 5896 tablet_copy_client.cc:647] T bddb4a31d81d44afaabc247fd1449767 P 28157bc3e678428fb5f93c077d528513: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:49.977998 5896 tablet_copy_client.cc:523] T bddb4a31d81d44afaabc247fd1449767 P 28157bc3e678428fb5f93c077d528513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 01:55:49.979801 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:55:49.983422 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: FlushMRSOp(d81f9cab671a4258aee1dd4ea4a5ee6b) complete. Timing: real 0.095s user 0.052s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":9306,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":960,"drs_written":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4,"lbm_write_time_us":897,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":15833,"threads_started":2,"wal-append.queue_time_us":6475}
I20240502 01:55:49.986032 5896 tablet_bootstrap.cc:492] T bddb4a31d81d44afaabc247fd1449767 P 28157bc3e678428fb5f93c077d528513: Bootstrap starting.
I20240502 01:55:49.986187 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling FlushMRSOp(3be66f4b90824a21a5f41915c9041970): perf score=0.033405
I20240502 01:55:49.992621 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: FlushMRSOp(fedbec8860904a2cb5294037b7a4bcad) complete. Timing: real 0.104s user 0.056s sys 0.031s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":16364,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":1589,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":902,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":42286,"threads_started":2,"wal-append.queue_time_us":26181}
I20240502 01:55:49.994839 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling UndoDeltaBlockGCOp(fedbec8860904a2cb5294037b7a4bcad): 4834 bytes on disk
I20240502 01:55:49.995479 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: UndoDeltaBlockGCOp(fedbec8860904a2cb5294037b7a4bcad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
W20240502 01:55:49.999090 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:55:50.023099 5896 tablet_bootstrap.cc:492] T bddb4a31d81d44afaabc247fd1449767 P 28157bc3e678428fb5f93c077d528513: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:55:50.033519 5896 tablet_bootstrap.cc:492] T bddb4a31d81d44afaabc247fd1449767 P 28157bc3e678428fb5f93c077d528513: Bootstrap complete.
I20240502 01:55:50.033948 5896 ts_tablet_manager.cc:1367] T bddb4a31d81d44afaabc247fd1449767 P 28157bc3e678428fb5f93c077d528513: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.003s
I20240502 01:55:50.035008 5896 raft_consensus.cc:348] T bddb4a31d81d44afaabc247fd1449767 P 28157bc3e678428fb5f93c077d528513 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
W20240502 01:55:50.038017 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:55:50.039211 5896 raft_consensus.cc:729] T bddb4a31d81d44afaabc247fd1449767 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28157bc3e678428fb5f93c077d528513, State: Initialized, Role: FOLLOWER
I20240502 01:55:50.039489 5896 consensus_queue.cc:260] T bddb4a31d81d44afaabc247fd1449767 P 28157bc3e678428fb5f93c077d528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:50.040387 5896 ts_tablet_manager.cc:1398] T bddb4a31d81d44afaabc247fd1449767 P 28157bc3e678428fb5f93c077d528513: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 01:55:50.040997 5891 tablet_bootstrap.cc:492] T de3eb3ed981d4f629eaa7f473ea3b430 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:50.041460 5891 tablet_bootstrap.cc:492] T de3eb3ed981d4f629eaa7f473ea3b430 P 0c64d40b5d7644bd8b29feaa80eab72f: Bootstrap complete.
I20240502 01:55:50.041801 5891 ts_tablet_manager.cc:1367] T de3eb3ed981d4f629eaa7f473ea3b430 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent bootstrapping tablet: real 0.078s user 0.043s sys 0.004s
I20240502 01:55:50.041791 22010 tablet_copy_service.cc:342] P 06543b05a16f42099bc86db2a0284412: Request end of tablet copy session 28157bc3e678428fb5f93c077d528513-bddb4a31d81d44afaabc247fd1449767 received from {username='slave'} at 127.14.212.198:53247
I20240502 01:55:50.042754 5891 raft_consensus.cc:348] T de3eb3ed981d4f629eaa7f473ea3b430 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:50.042974 5891 raft_consensus.cc:729] T de3eb3ed981d4f629eaa7f473ea3b430 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Initialized, Role: FOLLOWER
I20240502 01:55:50.043213 5891 consensus_queue.cc:260] T de3eb3ed981d4f629eaa7f473ea3b430 P 0c64d40b5d7644bd8b29feaa80eab72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:50.043787 5891 ts_tablet_manager.cc:1398] T de3eb3ed981d4f629eaa7f473ea3b430 P 0c64d40b5d7644bd8b29feaa80eab72f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:55:50.045235 22271 tablet_copy_service.cc:342] P c2a28baa7ee2469294d94b7fe5455cff: Request end of tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-de3eb3ed981d4f629eaa7f473ea3b430 received from {username='slave'} at 127.14.212.197:36211
I20240502 01:55:50.045358 22271 tablet_copy_service.cc:433] P c2a28baa7ee2469294d94b7fe5455cff: ending tablet copy session 0c64d40b5d7644bd8b29feaa80eab72f-de3eb3ed981d4f629eaa7f473ea3b430 on tablet de3eb3ed981d4f629eaa7f473ea3b430 with peer 0c64d40b5d7644bd8b29feaa80eab72f
I20240502 01:55:50.041908 22010 tablet_copy_service.cc:433] P 06543b05a16f42099bc86db2a0284412: ending tablet copy session 28157bc3e678428fb5f93c077d528513-bddb4a31d81d44afaabc247fd1449767 on tablet bddb4a31d81d44afaabc247fd1449767 with peer 28157bc3e678428fb5f93c077d528513
I20240502 01:55:50.066027 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: FlushMRSOp(3be66f4b90824a21a5f41915c9041970) complete. Timing: real 0.080s user 0.056s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":247,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":2952,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1520,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":255,"threads_started":1,"wal-append.queue_time_us":363}
I20240502 01:55:50.069605 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling UndoDeltaBlockGCOp(d81f9cab671a4258aee1dd4ea4a5ee6b): 4919 bytes on disk
I20240502 01:55:50.070322 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: UndoDeltaBlockGCOp(d81f9cab671a4258aee1dd4ea4a5ee6b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240502 01:55:50.072206 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling UndoDeltaBlockGCOp(3be66f4b90824a21a5f41915c9041970): 4581 bytes on disk
I20240502 01:55:50.073210 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: UndoDeltaBlockGCOp(3be66f4b90824a21a5f41915c9041970) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
W20240502 01:55:50.074051 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:50.142675 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:50.225694 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:50.309974 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:55:50.324049 24603 raft_consensus.cc:1203] T de3eb3ed981d4f629eaa7f473ea3b430 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:55:50.339727 24733 raft_consensus.cc:1203] T bddb4a31d81d44afaabc247fd1449767 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:55:50.340782 5883 raft_consensus.cc:2932] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:50.348594 24733 raft_consensus.cc:2932] T bddb4a31d81d44afaabc247fd1449767 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:50.353811 24603 raft_consensus.cc:2932] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:50.367973 21713 catalog_manager.cc:5484] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: config changed from index -1 to 2, VOTER 28157bc3e678428fb5f93c077d528513 (127.14.212.198) added. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
W20240502 01:55:50.415288 24669 consensus_peers.cc:485] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:50.436300 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:50.494557 24669 consensus_peers.cc:485] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:50.544991 24669 consensus_peers.cc:485] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:50.569111 24669 consensus_peers.cc:485] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:50.631032 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:50.739938 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:50.764019 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:50.775223 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:50.775194 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:50.786769 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240502 01:55:50.835947 22184 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.193:33785, user_credentials={real_user=slave}} blocked reactor thread for 49084.7us
I20240502 01:55:50.849128 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 01:55:50.849130 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:50.901257 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:55:50.915627 24583 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:50.960227 24715 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 01:55:51.025866 22053 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.194:44411, user_credentials={real_user=slave}} blocked reactor thread for 81682.9us
W20240502 01:55:51.038072 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:51.042388 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:55:51.102057 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:51.286335 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:51.321007 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:51.407744 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:51.420071 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:51.445884 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:55:51.447925 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:51.452999 24583 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:51.454459 24715 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 01:55:51.468730 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:51.495361 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:55:51.513335 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 01:55:51.545554 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:51.597431 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 01:55:51.598802 24669 consensus_peers.cc:485] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:55:51.599207 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:51.630612 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:51.635030 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:51.662994 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:51.672335 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:51.672473 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:55:51.676138 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240502 01:55:51.743690 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:51.786978 24669 consensus_peers.cc:485] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:55:51.789120 22248 consensus_queue.cc:237] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
W20240502 01:55:51.789723 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:55:51.791357 24601 raft_consensus.cc:1261] T de3eb3ed981d4f629eaa7f473ea3b430 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:51.791529 21988 raft_consensus.cc:1261] T de3eb3ed981d4f629eaa7f473ea3b430 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer c2a28baa7ee2469294d94b7fe5455cff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:51.791903 5975 consensus_queue.cc:1035] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:51.792181 5644 consensus_queue.cc:1035] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:51.795652 5644 raft_consensus.cc:2932] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:51.796547 21988 raft_consensus.cc:2932] T de3eb3ed981d4f629eaa7f473ea3b430 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:51.798377 5975 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:51.798537 5975 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:51.799340 5975 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:55:51.801218 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:55:51.801549 21854 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
W20240502 01:55:51.802534 22185 leader_election.cc:334] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:51.802803 22185 leader_election.cc:341] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 36d358202b544e74bcc378060082df4d
I20240502 01:55:51.802889 22185 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:51.803107 5975 raft_consensus.cc:2726] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:51.815150 21710 catalog_manager.cc:5484] T de3eb3ed981d4f629eaa7f473ea3b430 P 06543b05a16f42099bc86db2a0284412 reported cstate change: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New cstate: current_term: 1 leader_uuid: "c2a28baa7ee2469294d94b7fe5455cff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
I20240502 01:55:51.821029 24601 raft_consensus.cc:2932] T de3eb3ed981d4f629eaa7f473ea3b430 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } }
W20240502 01:55:51.841039 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:51.849272 24669 consensus_peers.cc:485] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:51.852283 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:51.890638 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:51.895321 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:51.904688 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:55:51.908548 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet de3eb3ed981d4f629eaa7f473ea3b430 with delete_type TABLET_DATA_TOMBSTONED (TS c64840657cdf4b93980bd34120c94d9a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40366
I20240502 01:55:51.909255 6089 tablet_replica.cc:330] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a: stopping tablet replica
I20240502 01:55:51.909479 6089 raft_consensus.cc:2229] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:51.909766 6089 raft_consensus.cc:2258] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:51.910465 6089 ts_tablet_manager.cc:1875] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:51.919862 6089 ts_tablet_manager.cc:1888] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:55:51.919984 6089 log.cc:1196] T de3eb3ed981d4f629eaa7f473ea3b430 P c64840657cdf4b93980bd34120c94d9a: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/wals/de3eb3ed981d4f629eaa7f473ea3b430
I20240502 01:55:51.921167 21701 catalog_manager.cc:4830] TS c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): tablet de3eb3ed981d4f629eaa7f473ea3b430 (table rebalance_test_table_15 [id=855af8637536470bb3f01aeac50e07e8]) successfully deleted
W20240502 01:55:51.921516 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:51.964982 24669 consensus_peers.cc:485] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:51.994894 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:52.072701 24669 consensus_peers.cc:485] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:52.083340 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:52.111518 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:52.119015 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:52.152024 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:55:52.230055 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:52.258662 24583 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:52.265117 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:52.270285 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling FlushMRSOp(5b3f40a9f1c34644b787905c28a54b29): perf score=0.033355
I20240502 01:55:52.297700 24715 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:52.313474 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:52.341681 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240502 01:55:52.356521 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:52.365789 24669 consensus_peers.cc:485] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:55:52.368785 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling FlushMRSOp(5b3f40a9f1c34644b787905c28a54b29): perf score=0.033390
I20240502 01:55:52.376749 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling FlushMRSOp(de3eb3ed981d4f629eaa7f473ea3b430): perf score=0.033380
I20240502 01:55:52.388751 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling FlushMRSOp(de3eb3ed981d4f629eaa7f473ea3b430): perf score=0.033393
W20240502 01:55:52.401819 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:52.422834 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:55:52.435760 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 01:55:52.443413 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:52.471446 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:52.473913 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:55:52.477926 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: FlushMRSOp(5b3f40a9f1c34644b787905c28a54b29) complete. Timing: real 0.207s user 0.085s sys 0.069s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":132261,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1732,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":931,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":139441,"threads_started":2,"wal-append.queue_time_us":7217}
I20240502 01:55:52.480434 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling FlushMRSOp(daaadb773202454894890c3379c4a58d): perf score=0.033409
I20240502 01:55:52.566232 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: FlushMRSOp(de3eb3ed981d4f629eaa7f473ea3b430) complete. Timing: real 0.189s user 0.030s sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":10921,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":27698,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1061,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":11048,"threads_started":1}
I20240502 01:55:52.569160 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling UndoDeltaBlockGCOp(de3eb3ed981d4f629eaa7f473ea3b430): 4581 bytes on disk
I20240502 01:55:52.569808 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: UndoDeltaBlockGCOp(de3eb3ed981d4f629eaa7f473ea3b430) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
W20240502 01:55:52.596100 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:52.601457 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: FlushMRSOp(daaadb773202454894890c3379c4a58d) complete. Timing: real 0.121s user 0.037s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":163,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1308,"drs_written":1,"lbm_read_time_us":209,"lbm_reads_lt_1ms":4,"lbm_write_time_us":921,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":22203,"threads_started":1,"wal-append.queue_time_us":20333}
I20240502 01:55:52.603973 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling UndoDeltaBlockGCOp(5b3f40a9f1c34644b787905c28a54b29): 4921 bytes on disk
I20240502 01:55:52.604955 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: UndoDeltaBlockGCOp(5b3f40a9f1c34644b787905c28a54b29) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4}
I20240502 01:55:52.607584 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling UndoDeltaBlockGCOp(daaadb773202454894890c3379c4a58d): 4834 bytes on disk
I20240502 01:55:52.608357 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: UndoDeltaBlockGCOp(daaadb773202454894890c3379c4a58d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240502 01:55:52.622320 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: FlushMRSOp(5b3f40a9f1c34644b787905c28a54b29) complete. Timing: real 0.251s user 0.033s sys 0.026s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8422,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":1526,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1092,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":258816,"thread_start_us":15214,"threads_started":2,"wal-append.queue_time_us":6390}
I20240502 01:55:52.625154 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling UndoDeltaBlockGCOp(5b3f40a9f1c34644b787905c28a54b29): 4921 bytes on disk
I20240502 01:55:52.627326 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: UndoDeltaBlockGCOp(5b3f40a9f1c34644b787905c28a54b29) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4}
W20240502 01:55:52.657354 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:55:52.681088 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: FlushMRSOp(de3eb3ed981d4f629eaa7f473ea3b430) complete. Timing: real 0.290s user 0.057s sys 0.018s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":5362,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1745,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":20678,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":635,"thread_start_us":5705,"threads_started":1}
I20240502 01:55:52.683781 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling UndoDeltaBlockGCOp(de3eb3ed981d4f629eaa7f473ea3b430): 4581 bytes on disk
I20240502 01:55:52.684469 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: UndoDeltaBlockGCOp(de3eb3ed981d4f629eaa7f473ea3b430) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240502 01:55:52.687578 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling FlushMRSOp(5b3f40a9f1c34644b787905c28a54b29): perf score=0.033471
W20240502 01:55:52.709676 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:52.732934 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:55:52.743896 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: FlushMRSOp(5b3f40a9f1c34644b787905c28a54b29) complete. Timing: real 0.056s user 0.048s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1418,"drs_written":1,"lbm_read_time_us":193,"lbm_reads_lt_1ms":4,"lbm_write_time_us":845,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6565,"threads_started":1,"wal-append.queue_time_us":6457}
I20240502 01:55:52.746922 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling UndoDeltaBlockGCOp(5b3f40a9f1c34644b787905c28a54b29): 4921 bytes on disk
I20240502 01:55:52.747599 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: UndoDeltaBlockGCOp(5b3f40a9f1c34644b787905c28a54b29) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
W20240502 01:55:52.757529 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:52.867652 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:52.889065 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:55:53.016006 24583 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:53.029161 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:53.040165 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 01:55:53.041131 24669 consensus_peers.cc:485] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:53.048622 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:53.062564 24715 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:53.063493 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 01:55:53.066442 24669 consensus_peers.cc:485] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:53.115803 24669 consensus_peers.cc:485] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:55:53.148408 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:53.148520 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:53.161041 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:55:53.280637 21857 consensus_queue.cc:237] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:53.282855 21988 raft_consensus.cc:1261] T ae04b73f8a574264a26a647400a44235 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:53.283154 24735 raft_consensus.cc:1261] T ae04b73f8a574264a26a647400a44235 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 759257ea7bef415ca698e60d82199d77: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:53.283430 5934 consensus_queue.cc:1035] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:53.283748 6197 consensus_queue.cc:1035] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:53.287319 5934 raft_consensus.cc:2932] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:53.287755 21988 raft_consensus.cc:2932] T ae04b73f8a574264a26a647400a44235 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:53.288205 24735 raft_consensus.cc:2932] T ae04b73f8a574264a26a647400a44235 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:53.294997 21706 catalog_manager.cc:5484] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 reported cstate change: config changed from index 2 to 3, VOTER 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200) evicted. New cstate: current_term: 1 leader_uuid: "759257ea7bef415ca698e60d82199d77" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:53.313382 24981 tablet_service.cc:1514] Processing DeleteTablet for tablet ae04b73f8a574264a26a647400a44235 with delete_type TABLET_DATA_TOMBSTONED (TS 41c70f373c9d49a49891b03a12c6cb2b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:39964
I20240502 01:55:53.315621 6243 tablet_replica.cc:330] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b: stopping tablet replica
I20240502 01:55:53.315904 6243 raft_consensus.cc:2229] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:53.316370 6243 raft_consensus.cc:2258] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:53.317203 6243 ts_tablet_manager.cc:1875] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:53.326447 6243 ts_tablet_manager.cc:1888] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 01:55:53.333137 6243 log.cc:1196] T ae04b73f8a574264a26a647400a44235 P 41c70f373c9d49a49891b03a12c6cb2b: Deleting WAL directory at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/wals/ae04b73f8a574264a26a647400a44235
I20240502 01:55:53.336122 21702 catalog_manager.cc:4830] TS 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097): tablet ae04b73f8a574264a26a647400a44235 (table rebalancer_extra_table_44 [id=75c3f22aa26848b68c2a3c2b5a6a5b3d]) successfully deleted
W20240502 01:55:53.371002 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:53.404641 24669 consensus_peers.cc:485] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:55:53.423296 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:53.456702 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:53.546484 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:53.561326 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:55:53.634311 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:53.660496 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:53.661798 24715 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:53.689955 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:53.701301 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:53.709923 24583 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:53.717182 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:55:53.737756 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240502 01:55:53.774045 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:53.785414 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:53.801224 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:53.827121 24669 consensus_peers.cc:485] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:53.896229 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:53.925307 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:53.984503 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:54.004098 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:54.029286 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:54.091243 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:54.093788 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:54.128010 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:54.134341 24669 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 01:55:54.135473 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:54.139876 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:54.153838 21923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 01:55:54.157239 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:55:54.163422 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:55:54.208451 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:54.242404 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:55:54.243453 24715 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 01:55:54.244946 21791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 01:55:54.250907 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:55:54.259356 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 01:55:54.272387 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:54.280360 22185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 104 similar messages]
W20240502 01:55:54.282073 24669 consensus_peers.cc:485] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:55:54.286816 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240502 01:55:54.288133 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:55:54.306229 24583 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:54.309204 24537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 01:55:54.356918 22054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 01:55:54.357648 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:55:54.369069 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:54.374389 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:54.383436 24934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 01:55:54.387073 24669 consensus_peers.cc:485] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:55:54.394065 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 01:55:54.416585 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:54.472568 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:54.515169 24669 consensus_peers.cc:485] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:54.543013 24669 consensus_peers.cc:485] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:54.654888 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:54.682166 24669 consensus_peers.cc:485] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:54.704963 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:54.704923 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:54.739212 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:55:54.813884 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:54.844385 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling FlushMRSOp(ad3cd6fff977431b9cd5df6d083bef80): perf score=0.033336
I20240502 01:55:54.846334 24583 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:54.856524 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:54.885864 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling FlushMRSOp(d52f4b5c815f4188a09eb12ec5375361): perf score=0.033361
I20240502 01:55:54.896072 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:54.904256 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling FlushMRSOp(d52f4b5c815f4188a09eb12ec5375361): perf score=0.033349
I20240502 01:55:54.905862 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:54.917078 24715 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 01:55:54.950851 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:55:54.958050 5140 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:54.958268 5140 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:54.959234 5140 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:55:54.961315 22248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:55:54.961547 22248 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
W20240502 01:55:54.962675 21791 leader_election.cc:341] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 36d358202b544e74bcc378060082df4d
W20240502 01:55:54.965847 21791 leader_election.cc:334] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:54.965975 21791 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:54.967031 5140 raft_consensus.cc:2726] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:55:54.967377 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:54.971757 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:54.986043 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:54.996260 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:55:55.018723 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:55:55.025207 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling FlushMRSOp(0f8200b044ef44e1a0451baf2b1485d2): perf score=0.033403
I20240502 01:55:55.041254 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: FlushMRSOp(ad3cd6fff977431b9cd5df6d083bef80) complete. Timing: real 0.197s user 0.027s sys 0.025s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":408,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":2112,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":927,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":18950,"threads_started":2,"wal-append.queue_time_us":12632}
I20240502 01:55:55.044389 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling UndoDeltaBlockGCOp(ad3cd6fff977431b9cd5df6d083bef80): 4581 bytes on disk
I20240502 01:55:55.045354 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: UndoDeltaBlockGCOp(ad3cd6fff977431b9cd5df6d083bef80) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":145,"lbm_reads_lt_1ms":4}
W20240502 01:55:55.076156 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:55:55.132190 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: FlushMRSOp(d52f4b5c815f4188a09eb12ec5375361) complete. Timing: real 0.246s user 0.044s sys 0.052s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":488,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":3417,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":895,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":114238,"threads_started":2,"wal-append.queue_time_us":92780}
I20240502 01:55:55.138490 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling UndoDeltaBlockGCOp(d52f4b5c815f4188a09eb12ec5375361): 4921 bytes on disk
I20240502 01:55:55.141705 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: UndoDeltaBlockGCOp(d52f4b5c815f4188a09eb12ec5375361) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
W20240502 01:55:55.145316 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:55:55.147289 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling FlushMRSOp(ad3cd6fff977431b9cd5df6d083bef80): perf score=0.033427
I20240502 01:55:55.215349 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: FlushMRSOp(d52f4b5c815f4188a09eb12ec5375361) complete. Timing: real 0.311s user 0.062s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":23423,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":4044,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":21402,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":32891,"threads_started":2,"wal-append.queue_time_us":9568}
I20240502 01:55:55.216506 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: FlushMRSOp(0f8200b044ef44e1a0451baf2b1485d2) complete. Timing: real 0.191s user 0.044s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":56124,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":3471,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1128,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":67455,"threads_started":2,"wal-append.queue_time_us":10541}
I20240502 01:55:55.218377 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling UndoDeltaBlockGCOp(d52f4b5c815f4188a09eb12ec5375361): 4921 bytes on disk
I20240502 01:55:55.219288 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling UndoDeltaBlockGCOp(0f8200b044ef44e1a0451baf2b1485d2): 4834 bytes on disk
I20240502 01:55:55.218983 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: UndoDeltaBlockGCOp(d52f4b5c815f4188a09eb12ec5375361) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240502 01:55:55.219715 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: FlushMRSOp(ad3cd6fff977431b9cd5df6d083bef80) complete. Timing: real 0.072s user 0.058s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1346,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":893,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":9022,"threads_started":1,"wal-append.queue_time_us":9129}
I20240502 01:55:55.220809 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: UndoDeltaBlockGCOp(0f8200b044ef44e1a0451baf2b1485d2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240502 01:55:55.222002 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling UndoDeltaBlockGCOp(ad3cd6fff977431b9cd5df6d083bef80): 4581 bytes on disk
I20240502 01:55:55.222657 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: UndoDeltaBlockGCOp(ad3cd6fff977431b9cd5df6d083bef80) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
W20240502 01:55:55.235754 24669 consensus_peers.cc:485] T a2c3475bcad94e3dad4be87376e500bc P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:55.282749 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:55.287065 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:55.445225 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:55.533900 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:55:55.553870 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:55.582636 24583 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:55.600713 24715 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:55:55.607167 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 01:55:55.630632 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:55:55.645018 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:55:55.646396 24669 consensus_peers.cc:485] T 61f537605bc746a0807e3c0a03ff2cc3 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:55.650051 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:55:55.675276 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 01:55:55.678804 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:55.691700 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240502 01:55:55.768425 24669 consensus_peers.cc:485] T 77a90f08d29a49d3b451f85cbdc828a6 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:55.931969 24669 consensus_peers.cc:485] T b60b948dae4841dd9c0919f2fcac5299 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:55.956817 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:55.998399 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:55:56.064288 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:55:56.156520 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:55:56.164484 24583 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:56.183977 21969 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:56.184459 24715 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 01:55:56.207757 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:55:56.215487 24981 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:55:56.254599 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 01:55:56.264217 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.275903 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:56.304543 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.346437 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.353826 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.353991 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:55:56.361245 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:55:56.367069 22100 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240502 01:55:56.393383 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.465981 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.488941 24669 consensus_peers.cc:485] T 97c9eb341ce1470db791324a882a0271 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.495082 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:55:56.513545 21989 consensus_queue.cc:237] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:56.524407 24999 raft_consensus.cc:1261] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:56.524394 24732 raft_consensus.cc:1261] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:55:56.524485 21924 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02a8e00ad2ad4bfa8bc1a21e50f821e4. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:56.525072 6528 consensus_queue.cc:1035] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:56.525375 6527 consensus_queue.cc:1035] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:55:56.531407 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:55:56.531744 6528 raft_consensus.cc:2932] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:56.532296 24999 raft_consensus.cc:2932] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:56.539733 21710 catalog_manager.cc:5484] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New cstate: current_term: 1 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
I20240502 01:55:56.543602 24732 raft_consensus.cc:2932] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 28157bc3e678428fb5f93c077d528513 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c64840657cdf4b93980bd34120c94d9a (127.14.212.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } }
W20240502 01:55:56.558152 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.571698 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.580992 21925 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.200:37097, user_credentials={real_user=slave}} blocked reactor thread for 44637.6us
W20240502 01:55:56.589648 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.637010 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.638028 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:55:56.650380 21988 consensus_queue.cc:237] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
W20240502 01:55:56.653612 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db96ccbe81564bb589b0de65977b3321. This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:56.653769 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:56.654017 21854 raft_consensus.cc:1261] T db96ccbe81564bb589b0de65977b3321 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:56.655541 6527 consensus_queue.cc:1035] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:55:56.661341 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.670078 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.671554 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.733672 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:56.740031 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.752689 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.909739 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.939307 24669 consensus_peers.cc:485] T 0a1b190978514cb49efb2b97cc97a65f P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.967446 24669 consensus_peers.cc:485] T 5a4a927b8f1740208f99ab1aca81b67a P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.981804 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:56.986454 24669 consensus_peers.cc:485] T 8b377c7402194b42a8eabc747f668fb2 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:57.043062 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:57.050629 24669 consensus_peers.cc:485] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:55:57.072535 6647 ts_tablet_manager.cc:902] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P c64840657cdf4b93980bd34120c94d9a: Initiating tablet copy from peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:55:57.073717 6647 tablet_copy_client.cc:308] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.194:44411
I20240502 01:55:57.074393 22010 tablet_copy_service.cc:140] P 06543b05a16f42099bc86db2a0284412: Received BeginTabletCopySession request for tablet 02a8e00ad2ad4bfa8bc1a21e50f821e4 from peer c64840657cdf4b93980bd34120c94d9a ({username='slave'} at 127.14.212.195:42095)
I20240502 01:55:57.074548 22010 tablet_copy_service.cc:161] P 06543b05a16f42099bc86db2a0284412: Beginning new tablet copy session on tablet 02a8e00ad2ad4bfa8bc1a21e50f821e4 from peer c64840657cdf4b93980bd34120c94d9a at {username='slave'} at 127.14.212.195:42095: session id = c64840657cdf4b93980bd34120c94d9a-02a8e00ad2ad4bfa8bc1a21e50f821e4
I20240502 01:55:57.076947 22010 tablet_copy_source_session.cc:205] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:57.079080 6647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a8e00ad2ad4bfa8bc1a21e50f821e4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:55:57.079988 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:55:57.084699 6647 tablet_copy_client.cc:783] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:57.084939 6647 tablet_copy_client.cc:647] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:57.087010 6647 tablet_copy_client.cc:523] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P c64840657cdf4b93980bd34120c94d9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:57.090016 6647 tablet_bootstrap.cc:492] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P c64840657cdf4b93980bd34120c94d9a: Bootstrap starting.
I20240502 01:55:57.095464 6647 tablet_bootstrap.cc:492] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P c64840657cdf4b93980bd34120c94d9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:57.095782 6647 tablet_bootstrap.cc:492] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P c64840657cdf4b93980bd34120c94d9a: Bootstrap complete.
I20240502 01:55:57.096032 6647 ts_tablet_manager.cc:1367] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P c64840657cdf4b93980bd34120c94d9a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240502 01:55:57.096791 6647 raft_consensus.cc:348] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:57.097043 6647 raft_consensus.cc:729] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Initialized, Role: FOLLOWER
I20240502 01:55:57.097254 6647 consensus_queue.cc:260] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P c64840657cdf4b93980bd34120c94d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:57.097788 6647 ts_tablet_manager.cc:1398] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P c64840657cdf4b93980bd34120c94d9a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:55:57.101809 22010 tablet_copy_service.cc:342] P 06543b05a16f42099bc86db2a0284412: Request end of tablet copy session c64840657cdf4b93980bd34120c94d9a-02a8e00ad2ad4bfa8bc1a21e50f821e4 received from {username='slave'} at 127.14.212.195:42095
I20240502 01:55:57.101936 22010 tablet_copy_service.cc:433] P 06543b05a16f42099bc86db2a0284412: ending tablet copy session c64840657cdf4b93980bd34120c94d9a-02a8e00ad2ad4bfa8bc1a21e50f821e4 on tablet 02a8e00ad2ad4bfa8bc1a21e50f821e4 with peer c64840657cdf4b93980bd34120c94d9a
W20240502 01:55:57.106004 24669 consensus_peers.cc:485] T 45ccb387e833453a950b1ef5b5282d29 P 28157bc3e678428fb5f93c077d528513 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:57.133850 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:57.232755 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:57.266165 6657 ts_tablet_manager.cc:902] T db96ccbe81564bb589b0de65977b3321 P 28157bc3e678428fb5f93c077d528513: Initiating tablet copy from peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
F20240502 01:55:57.266494 6657 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0eed9ee06a494588a32b4c951d6d3a14.queue_time_us, tablet-download-pool-0eed9ee06a494588a32b4c951d6d3a14.queue_time_us) (tablet-download-pool-0eed9ee06a494588a32b4c951d6d3a14.run_cpu_time_us, tablet-download-pool-0eed9ee06a494588a32b4c951d6d3a14.run_cpu_time_us) (tablet-download-pool-0eed9ee06a494588a32b4c951d6d3a14.run_wall_time_us, tablet-download-pool-0eed9ee06a494588a32b4c951d6d3a14.run_wall_time_us) (tablet-download-pool-0fb2e5d929334665ae41e33056c520bc.queue_time_us, tablet-download-pool-0fb2e5d929334665ae41e33056c520bc.queue_time_us) (tablet-download-pool-0fb2e5d929334665ae41e33056c520bc.run_cpu_time_us, tablet-download-pool-0fb2e5d929334665ae41e33056c520bc.run_cpu_time_us) (tablet-download-pool-0fb2e5d929334665ae41e33056c520bc.run_wall_time_us, tablet-download-pool-0fb2e5d929334665ae41e33056c520bc.run_wall_time_us) (tablet-download-pool-166e717fb9624ae58adb6699790daf7a.queue_time_us, tablet-download-pool-166e717fb9624ae58adb6699790daf7a.queue_time_us) (tablet-download-pool-166e717fb9624ae58adb6699790daf7a.run_cpu_time_us, tablet-download-pool-166e717fb9624ae58adb6699790daf7a.run_cpu_time_us) (tablet-download-pool-166e717fb9624ae58adb6699790daf7a.run_wall_time_us, tablet-download-pool-166e717fb9624ae58adb6699790daf7a.run_wall_time_us) (tablet-download-pool-2036e577d94242728ae4d00fe845121b.queue_time_us, tablet-download-pool-2036e577d94242728ae4d00fe845121b.queue_time_us) (tablet-download-pool-2036e577d94242728ae4d00fe845121b.run_cpu_time_us, tablet-download-pool-2036e577d94242728ae4d00fe845121b.run_cpu_time_us) (tablet-download-pool-2036e577d94242728ae4d00fe845121b.run_wall_time_us, tablet-download-pool-2036e577d94242728ae4d00fe845121b.run_wall_time_us) (tablet-download-pool-3a2f4ea104d4409383e5eb32ca2f78b0.queue_time_us, tablet-download-pool-3a2f4ea104d4409383e5eb32ca2f78b0.queue_time_us) (tablet-download-pool-3a2f4ea104d4409383e5eb32ca2f78b0.run_cpu_time_us, tablet-download-pool-3a2f4ea104d4409383e5eb32ca2f78b0.run_cpu_time_us) (tablet-download-pool-3a2f4ea104d4409383e5eb32ca2f78b0.run_wall_time_us, tablet-download-pool-3a2f4ea104d4409383e5eb32ca2f78b0.run_wall_time_us) (tablet-download-pool-3be66f4b90824a21a5f41915c9041970.queue_time_us, tablet-download-pool-3be66f4b90824a21a5f41915c9041970.queue_time_us) (tablet-download-pool-3be66f4b90824a21a5f41915c9041970.run_cpu_time_us, tablet-download-pool-3be66f4b90824a21a5f41915c9041970.run_cpu_time_us) (tablet-download-pool-3be66f4b90824a21a5f41915c9041970.run_wall_time_us, tablet-download-pool-3be66f4b90824a21a5f41915c9041970.run_wall_time_us) (tablet-download-pool-766241df5ee24264a5cb7176e6c97b91.queue_time_us, tablet-download-pool-766241df5ee24264a5cb7176e6c97b91.queue_time_us) (tablet-download-pool-766241df5ee24264a5cb7176e6c97b91.run_cpu_time_us, tablet-download-pool-766241df5ee24264a5cb7176e6c97b91.run_cpu_time_us) (tablet-download-pool-766241df5ee24264a5cb7176e6c97b91.run_wall_time_us, tablet-download-pool-766241df5ee24264a5cb7176e6c97b91.run_wall_time_us) (tablet-download-pool-77a90f08d29a49d3b451f85cbdc828a6.queue_time_us, tablet-download-pool-77a90f08d29a49d3b451f85cbdc828a6.queue_time_us) (tablet-download-pool-77a90f08d29a49d3b451f85cbdc828a6.run_cpu_time_us, tablet-download-pool-77a90f08d29a49d3b451f85cbdc828a6.run_cpu_time_us) (tablet-download-pool-77a90f08d29a49d3b451f85cbdc828a6.run_wall_time_us, tablet-download-pool-77a90f08d29a49d3b451f85cbdc828a6.run_wall_time_us) (tablet-download-pool-8a0ae0d8e8784db9988b3469d318f7fb.queue_time_us, tablet-download-pool-8a0ae0d8e8784db9988b3469d318f7fb.queue_time_us) (tablet-download-pool-8a0ae0d8e8784db9988b3469d318f7fb.run_cpu_time_us, tablet-download-pool-8a0ae0d8e8784db9988b3469d318f7fb.run_cpu_time_us) (tablet-download-pool-8a0ae0d8e8784db9988b3469d318f7fb.run_wall_time_us, tablet-download-pool-8a0ae0d8e8784db9988b3469d318f7fb.run_wall_time_us) (tablet-download-pool-ae04b73f8a574264a26a647400a44235.queue_time_us, tablet-download-pool-ae04b73f8a574264a26a647400a44235.queue_time_us) (tablet-download-pool-ae04b73f8a574264a26a647400a44235.run_cpu_time_us, tablet-download-pool-ae04b73f8a574264a26a647400a44235.run_cpu_time_us) (tablet-download-pool-ae04b73f8a574264a26a647400a44235.run_wall_time_us, tablet-download-pool-ae04b73f8a574264a26a647400a44235.run_wall_time_us) (tablet-download-pool-bddb4a31d81d44afaabc247fd1449767.queue_time_us, tablet-download-pool-bddb4a31d81d44afaabc247fd1449767.queue_time_us) (tablet-download-pool-bddb4a31d81d44afaabc247fd1449767.run_cpu_time_us, tablet-download-pool-bddb4a31d81d44afaabc247fd1449767.run_cpu_time_us) (tablet-download-pool-bddb4a31d81d44afaabc247fd1449767.run_wall_time_us, tablet-download-pool-bddb4a31d81d44afaabc247fd1449767.run_wall_time_us) (tablet-download-pool-d52f4b5c815f4188a09eb12ec5375361.queue_time_us, tablet-download-pool-d52f4b5c815f4188a09eb12ec5375361.queue_time_us) (tablet-download-pool-d52f4b5c815f4188a09eb12ec5375361.run_cpu_time_us, tablet-download-pool-d52f4b5c815f4188a09eb12ec5375361.run_cpu_time_us) (tablet-download-pool-d52f4b5c815f4188a09eb12ec5375361.run_wall_time_us, tablet-download-pool-d52f4b5c815f4188a09eb12ec5375361.run_wall_time_us) (tablet-download-pool-daaadb773202454894890c3379c4a58d.queue_time_us, tablet-download-pool-daaadb773202454894890c3379c4a58d.queue_time_us) (tablet-download-pool-daaadb773202454894890c3379c4a58d.run_cpu_time_us, tablet-download-pool-daaadb773202454894890c3379c4a58d.run_cpu_time_us) (tablet-download-pool-daaadb773202454894890c3379c4a58d.run_wall_time_us, tablet-download-pool-daaadb773202454894890c3379c4a58d.run_wall_time_us) (tablet-download-pool-db96ccbe81564bb589b0de65977b3321.queue_time_us, tablet-download-pool-db96ccbe81564bb589b0de65977b3321.queue_time_us) (tablet-download-pool-f018db81d6214bd0ac12feaf54541fed.queue_time_us, tablet-download-pool-f018db81d6214bd0ac12feaf54541fed.queue_time_us) (tablet-download-pool-f018db81d6214bd0ac12feaf54541fed.run_cpu_time_us, tablet-download-pool-f018db81d6214bd0ac12feaf54541fed.run_cpu_time_us) (tablet-download-pool-f018db81d6214bd0ac12feaf54541fed.run_wall_time_us, tablet-download-pool-f018db81d6214bd0ac12feaf54541fed.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fdc24571dcd google::LogMessage::Fail() at ??:0
@ 0x7fdc24575b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fdc245717cc google::LogMessage::Flush() at ??:0
@ 0x7fdc24572f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdc272b2742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fdc2727ef8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fdc27278cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fdc366cb8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fdc366e3daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fdc3680e480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fdc36846a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fdc36829981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fdc3682970e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fdc27282b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdc2728619c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fdc27286019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fdc2726101e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdc2a5e66db start_thread at ??:0
@ 0x7fdc2342471f clone at ??:0
W20240502 01:55:57.391022 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:57.406281 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:57.464149 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:57.585354 21923 connection.cc:535] client connection to 127.14.212.198:42375 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:57.585837 22185 connection.cc:535] client connection to 127.14.212.198:42375 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:57.586781 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.586863 24934 connection.cc:535] client connection to 127.14.212.198:42375 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:57.586953 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:55:57.587044 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.587138 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.587198 24537 connection.cc:535] client connection to 127.14.212.198:42375 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:57.587376 21791 connection.cc:535] client connection to 127.14.212.198:42375 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:57.588462 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.588610 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.588713 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.589170 22054 connection.cc:535] client connection to 127.14.212.198:42375 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:57.589967 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.590157 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.590305 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.590902 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.591143 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.591292 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.591413 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.592331 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.592456 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.592523 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.594834 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.596895 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.610226 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:57.612841 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:57.618976 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.626430 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.628113 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.634989 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:57.646762 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.657132 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:57.664444 22118 raft_consensus.cc:1203] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 01:55:57.667320 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.674861 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:57.698666 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.727764 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:55:57.741367 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:57.743062 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.748006 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.761283 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.780516 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.852756 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.855387 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:57.923331 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:57.994943 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:58.041286 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:58.081210 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:55:58.162967 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:58.179173 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:55:58.216315 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:55:58.370079 6679 raft_consensus.cc:482] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.370345 6679 raft_consensus.cc:504] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:58.371073 6679 leader_election.cc:288] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:55:58.372502 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:55:58.372666 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:58.372774 22054 leader_election.cc:302] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:58.373123 6679 raft_consensus.cc:2726] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:55:58.393990 6680 raft_consensus.cc:482] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.394222 6680 raft_consensus.cc:504] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:58.394902 6680 leader_election.cc:288] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:55:58.396504 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:58.396687 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.396773 24537 leader_election.cc:302] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:58.397038 6680 raft_consensus.cc:2726] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:55:58.409442 6622 raft_consensus.cc:482] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.409667 6622 raft_consensus.cc:504] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:58.410252 6622 leader_election.cc:288] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:58.410828 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0f7b64459848019ae2b748dd1ad8dc" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
W20240502 01:55:58.412034 21923 leader_election.cc:334] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.412155 21923 leader_election.cc:302] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06543b05a16f42099bc86db2a0284412; no voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:55:58.412442 6622 raft_consensus.cc:2726] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:58.413430 6646 raft_consensus.cc:482] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.413637 6646 raft_consensus.cc:504] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:58.414268 6646 leader_election.cc:288] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:58.414680 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b73e5695b28424ba127d1f434d1e88c" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
W20240502 01:55:58.415825 21791 leader_election.cc:334] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.415946 21791 leader_election.cc:302] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513
I20240502 01:55:58.416182 6646 raft_consensus.cc:2726] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:55:58.428258 6123 raft_consensus.cc:482] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.428440 6123 raft_consensus.cc:504] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:58.429337 6123 leader_election.cc:288] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:55:58.430061 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c5ae723e7a43a0b91a743b0c192a67" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
W20240502 01:55:58.430732 24934 leader_election.cc:334] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.430874 24934 leader_election.cc:302] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 06543b05a16f42099bc86db2a0284412, 28157bc3e678428fb5f93c077d528513
I20240502 01:55:58.431124 6123 raft_consensus.cc:2726] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:55:58.448686 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:55:58.451357 6622 raft_consensus.cc:482] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.451551 6622 raft_consensus.cc:504] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:58.452255 6622 leader_election.cc:288] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:58.452755 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dad8caf4f9744eea7be75d0a4890f65" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
W20240502 01:55:58.454008 21923 leader_election.cc:334] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.454142 21923 leader_election.cc:302] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06543b05a16f42099bc86db2a0284412; no voters: 28157bc3e678428fb5f93c077d528513, 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:55:58.454417 6622 raft_consensus.cc:2726] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:58.471391 6677 raft_consensus.cc:482] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.471638 6677 raft_consensus.cc:504] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:58.472429 6677 leader_election.cc:288] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:58.472748 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752d84f8d70e4bed91d0ea547923f22f" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:55:58.473225 6682 raft_consensus.cc:482] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.473409 6682 raft_consensus.cc:504] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:58.474043 6682 leader_election.cc:288] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:55:58.474083 22185 leader_election.cc:334] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.474205 22185 leader_election.cc:302] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, 28157bc3e678428fb5f93c077d528513
I20240502 01:55:58.475180 6677 raft_consensus.cc:2726] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:58.475230 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c5ae723e7a43a0b91a743b0c192a67" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:55:58.475484 25001 raft_consensus.cc:2454] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 1.
W20240502 01:55:58.475924 21923 leader_election.cc:334] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.476078 21925 leader_election.cc:302] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 41c70f373c9d49a49891b03a12c6cb2b; no voters: 28157bc3e678428fb5f93c077d528513
I20240502 01:55:58.476416 6682 raft_consensus.cc:2781] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:58.476533 6682 raft_consensus.cc:482] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.476644 6682 raft_consensus.cc:3037] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:58.478859 6682 raft_consensus.cc:504] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:58.479525 6682 leader_election.cc:288] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 election: Requested vote from peers 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:58.479871 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c5ae723e7a43a0b91a743b0c192a67" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:55:58.480077 25001 raft_consensus.cc:3037] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:55:58.481806 21923 leader_election.cc:334] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.482950 25001 raft_consensus.cc:2454] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06543b05a16f42099bc86db2a0284412 in term 2.
I20240502 01:55:58.483333 21925 leader_election.cc:302] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 41c70f373c9d49a49891b03a12c6cb2b; no voters: 28157bc3e678428fb5f93c077d528513
I20240502 01:55:58.483592 6682 raft_consensus.cc:2781] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:58.483742 6682 raft_consensus.cc:686] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [term 2 LEADER]: Becoming Leader. State: Replica: 06543b05a16f42099bc86db2a0284412, State: Running, Role: LEADER
I20240502 01:55:58.484041 6682 consensus_queue.cc:237] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:58.488044 21711 catalog_manager.cc:5484] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 reported cstate change: term changed from 1 to 2, leader changed from 28157bc3e678428fb5f93c077d528513 (127.14.212.198) to 06543b05a16f42099bc86db2a0284412 (127.14.212.194). New cstate: current_term: 2 leader_uuid: "06543b05a16f42099bc86db2a0284412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:58.495672 6123 raft_consensus.cc:482] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.495875 6123 raft_consensus.cc:504] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:58.496605 6123 leader_election.cc:288] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:55:58.497366 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cce126a8ba64fb2b40c4ddd03982f19" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
W20240502 01:55:58.498456 24934 leader_election.cc:334] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.498595 24934 leader_election.cc:302] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 28157bc3e678428fb5f93c077d528513, c64840657cdf4b93980bd34120c94d9a
I20240502 01:55:58.498878 6123 raft_consensus.cc:2726] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:58.503192 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling FlushMRSOp(3a2f4ea104d4409383e5eb32ca2f78b0): perf score=0.033335
I20240502 01:55:58.504716 6682 raft_consensus.cc:482] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.505034 6682 raft_consensus.cc:504] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:58.505770 6682 leader_election.cc:288] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:55:58.507249 21923 leader_election.cc:334] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:58.507457 21923 leader_election.cc:334] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.507547 21923 leader_election.cc:302] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06543b05a16f42099bc86db2a0284412; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:58.507856 6682 raft_consensus.cc:2726] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:55:58.527545 6118 raft_consensus.cc:482] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.527758 6118 raft_consensus.cc:504] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:58.528681 6118 leader_election.cc:288] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:55:58.529367 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cce126a8ba64fb2b40c4ddd03982f19" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:55:58.529623 25001 raft_consensus.cc:2454] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:55:58.530175 22056 leader_election.cc:302] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, c64840657cdf4b93980bd34120c94d9a; no voters:
W20240502 01:55:58.530664 22054 leader_election.cc:334] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.531128 6118 raft_consensus.cc:2781] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:58.531250 6118 raft_consensus.cc:482] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.531360 6118 raft_consensus.cc:3037] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:55:58.531474 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:55:58.535068 6118 raft_consensus.cc:504] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:55:58.536214 6118 leader_election.cc:288] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:55:58.536317 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cce126a8ba64fb2b40c4ddd03982f19" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b"
I20240502 01:55:58.536554 25001 raft_consensus.cc:3037] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:58.538676 25001 raft_consensus.cc:2454] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 2.
I20240502 01:55:58.539528 22056 leader_election.cc:302] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b, c64840657cdf4b93980bd34120c94d9a; no voters:
I20240502 01:55:58.540724 6118 raft_consensus.cc:2781] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:58.540927 6118 raft_consensus.cc:686] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [term 2 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:55:58.541491 6118 consensus_queue.cc:237] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
W20240502 01:55:58.541771 22054 leader_election.cc:334] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.546846 6123 raft_consensus.cc:482] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.547044 6123 raft_consensus.cc:504] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
W20240502 01:55:58.549163 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:55:58.549891 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0f7b64459848019ae2b748dd1ad8dc" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:55:58.550197 21988 raft_consensus.cc:2454] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:55:58.550428 6123 leader_election.cc:288] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:58.551415 24933 leader_election.cc:302] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
W20240502 01:55:58.553654 24934 leader_election.cc:334] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.554042 6123 raft_consensus.cc:2781] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:58.554200 6123 raft_consensus.cc:482] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.554364 6123 raft_consensus.cc:3037] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:58.559006 6123 raft_consensus.cc:504] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:58.558939 21711 catalog_manager.cc:5484] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 1 to 2, leader changed from 28157bc3e678428fb5f93c077d528513 (127.14.212.198) to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 2 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:55:58.561201 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0f7b64459848019ae2b748dd1ad8dc" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:55:58.561460 21988 raft_consensus.cc:3037] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:58.561560 6123 leader_election.cc:288] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 election: Requested vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:55:58.563504 24934 leader_election.cc:334] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.564414 21988 raft_consensus.cc:2454] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 2.
I20240502 01:55:58.564975 24933 leader_election.cc:302] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 41c70f373c9d49a49891b03a12c6cb2b; no voters: 28157bc3e678428fb5f93c077d528513
I20240502 01:55:58.567559 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: FlushMRSOp(3a2f4ea104d4409383e5eb32ca2f78b0) complete. Timing: real 0.064s user 0.046s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":434,"dirs.run_cpu_time_us":272,"dirs.run_wall_time_us":7047,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":776,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":8385,"threads_started":2,"wal-append.queue_time_us":7608}
I20240502 01:55:58.568203 6123 raft_consensus.cc:2781] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:58.568338 6123 raft_consensus.cc:686] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [term 2 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:55:58.568634 6123 consensus_queue.cc:237] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:58.569329 6553 raft_consensus.cc:482] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.569492 6553 raft_consensus.cc:504] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:58.570211 6553 leader_election.cc:288] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:55:58.570184 22167 maintenance_manager.cc:419] P c64840657cdf4b93980bd34120c94d9a: Scheduling UndoDeltaBlockGCOp(3a2f4ea104d4409383e5eb32ca2f78b0): 4835 bytes on disk
I20240502 01:55:58.570820 22057 maintenance_manager.cc:643] P c64840657cdf4b93980bd34120c94d9a: UndoDeltaBlockGCOp(3a2f4ea104d4409383e5eb32ca2f78b0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240502 01:55:58.570926 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dad8caf4f9744eea7be75d0a4890f65" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:55:58.571156 21988 raft_consensus.cc:2454] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:55:58.571785 24933 leader_election.cc:302] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 41c70f373c9d49a49891b03a12c6cb2b; no voters:
I20240502 01:55:58.572325 6123 raft_consensus.cc:2781] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:58.572635 6123 raft_consensus.cc:482] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.572757 6123 raft_consensus.cc:3037] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:55:58.573336 24934 leader_election.cc:334] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.575497 6123 raft_consensus.cc:504] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:58.576187 21711 catalog_manager.cc:5484] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 1 to 2, leader changed from 28157bc3e678428fb5f93c077d528513 (127.14.212.198) to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 2 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:58.576467 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dad8caf4f9744eea7be75d0a4890f65" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:55:58.576709 21988 raft_consensus.cc:3037] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:55:58.577514 24934 leader_election.cc:334] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.579094 21988 raft_consensus.cc:2454] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 2.
I20240502 01:55:58.579627 24933 leader_election.cc:302] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, 41c70f373c9d49a49891b03a12c6cb2b; no voters: 28157bc3e678428fb5f93c077d528513
I20240502 01:55:58.579922 6553 raft_consensus.cc:2781] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:58.580222 6123 leader_election.cc:288] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:55:58.583657 6553 raft_consensus.cc:686] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:55:58.583951 6553 consensus_queue.cc:237] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:58.586297 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling FlushMRSOp(84d602fa045e4cfa9b698d3e92e5cf8f): perf score=0.033360
I20240502 01:55:58.600522 6646 raft_consensus.cc:482] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.600421 21711 catalog_manager.cc:5484] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 1 to 2, leader changed from 28157bc3e678428fb5f93c077d528513 (127.14.212.198) to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 2 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:58.600731 6646 raft_consensus.cc:504] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:58.601454 6646 leader_election.cc:288] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:55:58.605733 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:58.605964 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.606076 21791 leader_election.cc:302] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:58.607864 6646 raft_consensus.cc:2726] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:58.624173 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: FlushMRSOp(84d602fa045e4cfa9b698d3e92e5cf8f) complete. Timing: real 0.035s user 0.031s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":467,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":1599,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":761,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":591,"threads_started":2,"wal-append.queue_time_us":367}
I20240502 01:55:58.627040 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling UndoDeltaBlockGCOp(84d602fa045e4cfa9b698d3e92e5cf8f): 4921 bytes on disk
I20240502 01:55:58.627709 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: UndoDeltaBlockGCOp(84d602fa045e4cfa9b698d3e92e5cf8f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240502 01:55:58.629907 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling FlushMRSOp(3a2f4ea104d4409383e5eb32ca2f78b0): perf score=0.033362
W20240502 01:55:58.636854 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:55:58.639421 6682 raft_consensus.cc:482] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.639739 6682 raft_consensus.cc:504] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:58.641130 22118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c38c0987ed469eba1fb6f76ee2f6a9" candidate_uuid: "06543b05a16f42099bc86db2a0284412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c64840657cdf4b93980bd34120c94d9a" is_pre_election: true
I20240502 01:55:58.641973 6118 raft_consensus.cc:482] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.642138 6118 raft_consensus.cc:504] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:58.642874 6118 leader_election.cc:288] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:58.646675 6682 leader_election.cc:288] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
W20240502 01:55:58.653904 22054 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:58.654112 22054 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.654273 22054 leader_election.cc:302] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:58.654621 6118 raft_consensus.cc:2726] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:55:58.655408 21923 leader_election.cc:334] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.655573 21923 leader_election.cc:302] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06543b05a16f42099bc86db2a0284412; no voters: 28157bc3e678428fb5f93c077d528513, c64840657cdf4b93980bd34120c94d9a
I20240502 01:55:58.655912 6682 raft_consensus.cc:2726] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:58.658701 6589 raft_consensus.cc:482] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.658859 6589 raft_consensus.cc:504] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:58.659740 6589 leader_election.cc:288] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:58.660605 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752d84f8d70e4bed91d0ea547923f22f" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:55:58.660902 22250 raft_consensus.cc:2454] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 1.
I20240502 01:55:58.662114 24538 leader_election.cc:302] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:55:58.662786 6589 raft_consensus.cc:2781] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:58.662882 6589 raft_consensus.cc:482] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.662987 6589 raft_consensus.cc:3037] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:55:58.663313 24537 leader_election.cc:334] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.666692 6589 raft_consensus.cc:504] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:58.666867 6653 raft_consensus.cc:482] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.667068 6653 raft_consensus.cc:504] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:55:58.667801 6653 leader_election.cc:288] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:58.668138 6589 leader_election.cc:288] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: Requested vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:55:58.669167 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752d84f8d70e4bed91d0ea547923f22f" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff"
I20240502 01:55:58.669420 22250 raft_consensus.cc:3037] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:55:58.670183 22185 leader_election.cc:334] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:58.670390 22185 leader_election.cc:334] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.670480 22185 leader_election.cc:302] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:58.670725 6653 raft_consensus.cc:2726] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:58.672789 22250 raft_consensus.cc:2454] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 2.
I20240502 01:55:58.673374 24538 leader_election.cc:302] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, c2a28baa7ee2469294d94b7fe5455cff; no voters:
I20240502 01:55:58.673797 6589 raft_consensus.cc:2781] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:58.673965 6589 raft_consensus.cc:686] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:55:58.674363 6589 consensus_queue.cc:237] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
W20240502 01:55:58.676957 24537 leader_election.cc:334] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.677804 6118 raft_consensus.cc:482] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.677995 6118 raft_consensus.cc:504] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:58.678882 6118 leader_election.cc:288] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:55:58.680357 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: FlushMRSOp(3a2f4ea104d4409383e5eb32ca2f78b0) complete. Timing: real 0.050s user 0.033s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":3867,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":997,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":420,"threads_started":1,"wal-append.queue_time_us":551}
W20240502 01:55:58.680647 22054 leader_election.cc:334] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.681954 21711 catalog_manager.cc:5484] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 1 to 2, leader changed from 28157bc3e678428fb5f93c077d528513 (127.14.212.198) to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 2 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } }
I20240502 01:55:58.683174 22298 maintenance_manager.cc:419] P c2a28baa7ee2469294d94b7fe5455cff: Scheduling UndoDeltaBlockGCOp(3a2f4ea104d4409383e5eb32ca2f78b0): 4835 bytes on disk
I20240502 01:55:58.683825 22188 maintenance_manager.cc:643] P c2a28baa7ee2469294d94b7fe5455cff: UndoDeltaBlockGCOp(3a2f4ea104d4409383e5eb32ca2f78b0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240502 01:55:58.685920 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c38c0987ed469eba1fb6f76ee2f6a9" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412" is_pre_election: true
I20240502 01:55:58.686184 21988 raft_consensus.cc:2454] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 1.
I20240502 01:55:58.686604 22053 leader_election.cc:302] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513
I20240502 01:55:58.686964 6118 raft_consensus.cc:2781] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:55:58.687096 6118 raft_consensus.cc:482] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.687201 6118 raft_consensus.cc:3037] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:58.689402 6118 raft_consensus.cc:504] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:58.690059 6118 leader_election.cc:288] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:55:58.690618 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c38c0987ed469eba1fb6f76ee2f6a9" candidate_uuid: "c64840657cdf4b93980bd34120c94d9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "06543b05a16f42099bc86db2a0284412"
I20240502 01:55:58.690840 21988 raft_consensus.cc:3037] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:55:58.691529 22054 leader_election.cc:334] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.693428 21988 raft_consensus.cc:2454] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64840657cdf4b93980bd34120c94d9a in term 2.
I20240502 01:55:58.693819 22053 leader_election.cc:302] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513
I20240502 01:55:58.694168 6118 raft_consensus.cc:2781] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:58.694329 6118 raft_consensus.cc:686] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [term 2 LEADER]: Becoming Leader. State: Replica: c64840657cdf4b93980bd34120c94d9a, State: Running, Role: LEADER
I20240502 01:55:58.694615 6118 consensus_queue.cc:237] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:55:58.698591 21711 catalog_manager.cc:5484] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a reported cstate change: term changed from 1 to 2, leader changed from 28157bc3e678428fb5f93c077d528513 (127.14.212.198) to c64840657cdf4b93980bd34120c94d9a (127.14.212.195). New cstate: current_term: 2 leader_uuid: "c64840657cdf4b93980bd34120c94d9a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } }
I20240502 01:55:58.710994 6646 raft_consensus.cc:482] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.711239 6646 raft_consensus.cc:504] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:58.711877 6678 raft_consensus.cc:482] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.712007 6646 leader_election.cc:288] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:55:58.712060 6678 raft_consensus.cc:504] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:58.712672 6678 leader_election.cc:288] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:55:58.713634 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:55:58.714479 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:58.714592 21791 leader_election.cc:302] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:58.714792 6646 raft_consensus.cc:2726] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:55:58.714849 21791 leader_election.cc:334] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:58.715047 21791 leader_election.cc:334] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.715195 21791 leader_election.cc:302] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:58.715446 6646 raft_consensus.cc:2726] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:55:58.733798 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:58.738451 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:58.753659 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling FlushMRSOp(84d602fa045e4cfa9b698d3e92e5cf8f): perf score=0.033396
I20240502 01:55:58.754101 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling FlushMRSOp(d4b6dfe0e8da4352b136d6f97d473bad): perf score=0.033401
I20240502 01:55:58.764448 6680 raft_consensus.cc:482] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.764734 6680 raft_consensus.cc:504] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:58.766463 6680 leader_election.cc:288] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
W20240502 01:55:58.767140 24537 leader_election.cc:334] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:55:58.767371 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:55:58.767697 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b73e5695b28424ba127d1f434d1e88c" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:55:58.767977 21857 raft_consensus.cc:2454] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 2.
I20240502 01:55:58.768500 24536 leader_election.cc:302] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513
I20240502 01:55:58.769133 6680 raft_consensus.cc:2781] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 01:55:58.769256 6680 raft_consensus.cc:482] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.769359 6680 raft_consensus.cc:3037] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:55:58.772483 6680 raft_consensus.cc:504] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:58.774039 6680 leader_election.cc:288] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 election: Requested vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
W20240502 01:55:58.775578 24537 leader_election.cc:334] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:55:58.778752 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:55:58.778910 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b73e5695b28424ba127d1f434d1e88c" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77"
I20240502 01:55:58.779150 21857 raft_consensus.cc:3037] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Advancing to term 3
W20240502 01:55:58.781083 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:55:58.781263 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:55:58.782308 21857 raft_consensus.cc:2454] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c64d40b5d7644bd8b29feaa80eab72f in term 3.
I20240502 01:55:58.782964 24536 leader_election.cc:302] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513
I20240502 01:55:58.783959 6589 raft_consensus.cc:2781] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:55:58.787475 6589 raft_consensus.cc:686] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [term 3 LEADER]: Becoming Leader. State: Replica: 0c64d40b5d7644bd8b29feaa80eab72f, State: Running, Role: LEADER
I20240502 01:55:58.787954 6589 consensus_queue.cc:237] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:55:58.799113 21711 catalog_manager.cc:5484] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f reported cstate change: term changed from 2 to 3, leader changed from 28157bc3e678428fb5f93c077d528513 (127.14.212.198) to 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197). New cstate: current_term: 3 leader_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } }
I20240502 01:55:58.814908 6118 raft_consensus.cc:482] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.815128 6118 raft_consensus.cc:504] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:55:58.815984 6118 leader_election.cc:288] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:55:58.818234 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:58.819164 22054 leader_election.cc:334] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:58.819401 22054 leader_election.cc:334] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:58.819507 22054 leader_election.cc:302] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:58.820138 6118 raft_consensus.cc:2726] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:58.829702 6653 raft_consensus.cc:482] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:58.829910 6653 raft_consensus.cc:504] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:58.830636 6653 leader_election.cc:288] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:55:58.832325 22185 leader_election.cc:334] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:55:58.832535 22185 leader_election.cc:334] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:58.832643 22185 leader_election.cc:302] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:58.832926 6653 raft_consensus.cc:2726] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:58.835496 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: FlushMRSOp(84d602fa045e4cfa9b698d3e92e5cf8f) complete. Timing: real 0.082s user 0.040s sys 0.028s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":13920,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":8874,"drs_written":1,"lbm_read_time_us":138,"lbm_reads_lt_1ms":4,"lbm_write_time_us":959,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":16485,"threads_started":2,"wal-append.queue_time_us":2789}
I20240502 01:55:58.838214 21904 maintenance_manager.cc:419] P 759257ea7bef415ca698e60d82199d77: Scheduling UndoDeltaBlockGCOp(84d602fa045e4cfa9b698d3e92e5cf8f): 4921 bytes on disk
I20240502 01:55:58.838903 21794 maintenance_manager.cc:643] P 759257ea7bef415ca698e60d82199d77: UndoDeltaBlockGCOp(84d602fa045e4cfa9b698d3e92e5cf8f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4}
W20240502 01:55:58.840714 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:55:58.841580 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: FlushMRSOp(d4b6dfe0e8da4352b136d6f97d473bad) complete. Timing: real 0.087s user 0.055s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6651,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1859,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":766,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7664,"threads_started":2,"wal-append.queue_time_us":977}
I20240502 01:55:58.843951 22036 maintenance_manager.cc:419] P 06543b05a16f42099bc86db2a0284412: Scheduling UndoDeltaBlockGCOp(d4b6dfe0e8da4352b136d6f97d473bad): 4581 bytes on disk
I20240502 01:55:58.844460 21926 maintenance_manager.cc:643] P 06543b05a16f42099bc86db2a0284412: UndoDeltaBlockGCOp(d4b6dfe0e8da4352b136d6f97d473bad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240502 01:55:58.845584 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:58.869560 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:58.943053 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:58.947708 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:58.963933 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:58.968219 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:59.005544 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:59.024153 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:55:59.024446 24999 raft_consensus.cc:1261] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Refusing update from remote peer 06543b05a16f42099bc86db2a0284412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:55:59.025082 6682 consensus_queue.cc:1035] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:55:59.026968 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:59.033936 21988 raft_consensus.cc:1261] T 6dad8caf4f9744eea7be75d0a4890f65 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:55:59.034583 6694 consensus_queue.cc:1035] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:59.046703 6653 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:55:59.046869 6653 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:55:59.047667 6653 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:55:59.048082 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:55:59.048329 21857 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
W20240502 01:55:59.049496 22185 leader_election.cc:341] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 36d358202b544e74bcc378060082df4d
W20240502 01:55:59.049656 22185 leader_election.cc:334] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:59.049785 22185 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:59.050067 6653 raft_consensus.cc:2726] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:59.068274 22250 raft_consensus.cc:1261] T 752d84f8d70e4bed91d0ea547923f22f P c2a28baa7ee2469294d94b7fe5455cff [term 2 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:55:59.068946 6707 consensus_queue.cc:1035] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:55:59.076422 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:59.089102 24999 raft_consensus.cc:1261] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:55:59.089772 6118 consensus_queue.cc:1035] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:55:59.103612 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:59.106850 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:59.131156 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:59.134665 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:59.142818 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:55:59.144599 21988 raft_consensus.cc:1261] T cc0f7b64459848019ae2b748dd1ad8dc P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Refusing update from remote peer 41c70f373c9d49a49891b03a12c6cb2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:55:59.145239 6694 consensus_queue.cc:1035] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:55:59.160343 21989 raft_consensus.cc:1261] T 00c38c0987ed469eba1fb6f76ee2f6a9 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Refusing update from remote peer c64840657cdf4b93980bd34120c94d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:55:59.163023 6706 consensus_queue.cc:1035] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:55:59.172850 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:59.178541 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:59.186396 21923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240502 01:55:59.187871 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:59.198801 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:59.252863 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:59.253412 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:59.259886 21857 raft_consensus.cc:1261] T 0b73e5695b28424ba127d1f434d1e88c P 759257ea7bef415ca698e60d82199d77 [term 3 FOLLOWER]: Refusing update from remote peer 0c64d40b5d7644bd8b29feaa80eab72f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 01:55:59.260509 6707 consensus_queue.cc:1035] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:55:59.290151 22185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240502 01:55:59.291100 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:59.295126 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:59.303512 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:59.313062 24537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240502 01:55:59.325135 21791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240502 01:55:59.358886 22054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 01:55:59.383540 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:59.405153 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:59.423612 24934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 01:55:59.504519 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:59.631655 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:55:59.642555 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:59.716419 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:59.734057 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:59.745752 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:59.765343 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:59.820099 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:59.850975 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:59.865803 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:59.916443 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:55:59.920039 6589 raft_consensus.cc:482] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:55:59.920226 6589 raft_consensus.cc:504] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:55:59.920905 6589 leader_election.cc:288] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:55:59.923009 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:55:59.923203 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:55:59.923285 24537 leader_election.cc:302] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:55:59.923499 6589 raft_consensus.cc:2726] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:55:59.924520 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:55:59.926548 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:59.969877 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:59.976517 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.011087 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.020537 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.028784 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:00.056526 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.059046 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:00.062536 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.071352 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.083447 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.092125 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.093293 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.120168 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:00.135768 6682 raft_consensus.cc:482] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:00.135987 6682 raft_consensus.cc:504] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:00.136804 6682 leader_election.cc:288] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:00.138581 21923 leader_election.cc:334] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:00.138774 21923 leader_election.cc:334] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:00.138880 21923 leader_election.cc:302] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06543b05a16f42099bc86db2a0284412; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:00.139133 6682 raft_consensus.cc:2726] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:00.147019 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.150715 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.159202 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.159837 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.168092 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.182974 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.203224 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:00.226011 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.227087 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.245345 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.248073 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:00.261974 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:00.285176 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.287127 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:00.296289 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:56:00.312762 6706 raft_consensus.cc:482] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:00.313123 6706 raft_consensus.cc:504] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:56:00.314018 6706 leader_election.cc:288] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:56:00.316115 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:00.316329 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:00.316435 22054 leader_election.cc:302] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:00.316696 6706 raft_consensus.cc:2726] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:00.336309 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:00.380452 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:56:00.468920 6729 raft_consensus.cc:482] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:00.469180 6729 raft_consensus.cc:504] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:00.469913 6729 leader_election.cc:288] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:00.472502 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:00.472739 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:00.472858 21791 leader_election.cc:302] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:00.473213 6729 raft_consensus.cc:2726] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:00.496459 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:56:00.506539 6729 raft_consensus.cc:482] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:00.506783 6729 raft_consensus.cc:504] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:00.507642 6729 leader_election.cc:288] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:56:00.509443 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:00.509649 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:00.509784 21791 leader_election.cc:302] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:00.513360 6729 raft_consensus.cc:2726] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:00.521040 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:00.554102 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:56:00.571236 6118 raft_consensus.cc:482] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:00.571492 6118 raft_consensus.cc:504] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:00.572283 6118 leader_election.cc:288] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:00.574604 22054 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:00.574806 22054 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:00.574898 22054 leader_election.cc:302] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:00.575203 6118 raft_consensus.cc:2726] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:56:00.596807 6727 raft_consensus.cc:482] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:00.597023 6727 raft_consensus.cc:504] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:56:00.597620 6727 leader_election.cc:288] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:00.599123 22185 leader_election.cc:334] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:00.599303 22185 leader_election.cc:334] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:00.599380 22185 leader_election.cc:302] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:00.599656 6727 raft_consensus.cc:2726] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:56:00.619192 6118 raft_consensus.cc:482] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:00.619423 6118 raft_consensus.cc:504] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:56:00.620044 6118 leader_election.cc:288] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:00.621876 22054 leader_election.cc:334] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:00.622124 22054 leader_election.cc:334] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:00.622218 22054 leader_election.cc:302] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:00.622493 6118 raft_consensus.cc:2726] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:56:00.672850 6727 raft_consensus.cc:482] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:00.673101 6727 raft_consensus.cc:504] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:00.673780 6727 leader_election.cc:288] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:00.676152 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:00.684196 22185 leader_election.cc:334] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:00.684432 22185 leader_election.cc:334] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:00.684548 22185 leader_election.cc:302] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:00.684849 6727 raft_consensus.cc:2726] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:56:00.700717 6729 raft_consensus.cc:482] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:00.700978 6729 raft_consensus.cc:504] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
W20240502 01:56:00.701292 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:56:00.701783 6729 leader_election.cc:288] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:00.703657 21791 leader_election.cc:334] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:00.703864 21791 leader_election.cc:334] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:00.703974 21791 leader_election.cc:302] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:00.704260 6729 raft_consensus.cc:2726] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:00.802687 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:00.803028 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:00.808444 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:00.906522 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:01.004653 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.029320 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.181370 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:01.218899 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.255333 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.256528 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:01.323387 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:01.350051 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.359457 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.380306 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:01.381052 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:01.383958 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:01.421746 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:56:01.491421 6589 raft_consensus.cc:482] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:01.491634 6589 raft_consensus.cc:504] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:01.492394 6589 leader_election.cc:288] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:01.494009 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:01.494230 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:01.494350 24537 leader_election.cc:302] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:01.494661 6589 raft_consensus.cc:2726] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:01.515339 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.538748 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.550462 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.561805 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.567775 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:01.582581 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.597776 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.613193 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:01.621997 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.666290 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.704105 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.708742 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:01.726195 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:56:01.728335 6740 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:56:01.728543 6740 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:56:01.729698 6740 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:56:01.730085 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:56:01.730381 21854 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
W20240502 01:56:01.730922 22185 leader_election.cc:341] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 36d358202b544e74bcc378060082df4d
W20240502 01:56:01.731105 22185 leader_election.cc:334] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:01.731233 22185 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:01.731498 6740 raft_consensus.cc:2726] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:01.733355 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.753933 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.759850 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.770581 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.797040 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:01.827334 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:01.830247 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.832618 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.833362 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:01.839572 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:01.986915 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.050442 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:02.072650 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:02.097640 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:02.134435 6682 raft_consensus.cc:482] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:02.134619 6682 raft_consensus.cc:504] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:02.135248 6682 leader_election.cc:288] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:02.136687 21923 leader_election.cc:334] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:02.136909 21923 leader_election.cc:334] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:02.137009 21923 leader_election.cc:302] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06543b05a16f42099bc86db2a0284412; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:02.137251 6682 raft_consensus.cc:2726] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:56:02.170447 6653 raft_consensus.cc:482] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:02.170707 6653 raft_consensus.cc:504] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:56:02.171357 6653 leader_election.cc:288] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:02.172854 22185 leader_election.cc:334] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:02.173074 22185 leader_election.cc:334] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:02.173170 22185 leader_election.cc:302] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:02.173420 6653 raft_consensus.cc:2726] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:02.197470 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.228144 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.234948 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:02.274333 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:02.287745 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.296787 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:02.319017 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.338820 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:02.345639 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.356134 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.356639 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:02.360119 6740 raft_consensus.cc:482] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:02.360304 6740 raft_consensus.cc:504] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:02.361021 6740 leader_election.cc:288] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:02.365262 22185 leader_election.cc:334] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:02.365463 22185 leader_election.cc:334] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:02.365582 22185 leader_election.cc:302] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:02.365845 6740 raft_consensus.cc:2726] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:02.399427 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.439496 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.453346 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.461934 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.496418 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.513659 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.527418 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.545841 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:02.546108 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:02.576771 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.597497 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.598256 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.598965 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:02.622617 6742 raft_consensus.cc:482] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:02.622822 6742 raft_consensus.cc:504] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:02.623447 6742 leader_election.cc:288] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:02.625067 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:02.625289 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:02.625383 21791 leader_election.cc:302] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:02.625689 6742 raft_consensus.cc:2726] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:02.646164 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:02.670599 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:02.706437 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:02.709699 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:02.723626 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.739262 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:02.741827 6738 raft_consensus.cc:482] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:02.742020 6738 raft_consensus.cc:504] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:56:02.742725 6738 leader_election.cc:288] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:56:02.745011 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:02.745206 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:02.745306 22054 leader_election.cc:302] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:02.745606 6738 raft_consensus.cc:2726] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:02.754962 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.809468 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:02.810108 6729 raft_consensus.cc:482] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:02.810336 6729 raft_consensus.cc:504] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:02.811019 6729 leader_election.cc:288] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:56:02.814447 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:02.814668 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:02.814764 21791 leader_election.cc:302] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:02.815054 6729 raft_consensus.cc:2726] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:02.841596 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:02.844120 6729 raft_consensus.cc:482] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:02.844318 6729 raft_consensus.cc:504] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:02.844936 6729 leader_election.cc:288] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:02.846315 21791 leader_election.cc:334] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:02.846482 21791 leader_election.cc:334] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:02.846565 21791 leader_election.cc:302] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:02.846800 6729 raft_consensus.cc:2726] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:02.851817 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.863494 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.867425 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.883460 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:02.886436 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.920164 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:02.927130 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:03.005103 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:03.067886 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:03.100140 6747 raft_consensus.cc:482] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:03.100327 6747 raft_consensus.cc:504] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:03.101032 6747 leader_election.cc:288] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:03.103189 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:03.103399 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:03.103466 24537 leader_election.cc:302] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:03.103686 6731 raft_consensus.cc:2726] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:03.127333 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:03.154664 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:03.210799 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:03.257022 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:03.291090 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:03.486461 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:03.531821 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:03.537842 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:03.573990 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:03.600905 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:56:03.646911 6738 raft_consensus.cc:482] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:03.647118 6738 raft_consensus.cc:504] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:03.647729 6738 leader_election.cc:288] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:03.649705 22054 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:03.649957 22054 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:03.650103 22054 leader_election.cc:302] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:03.650439 6738 raft_consensus.cc:2726] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:56:03.758998 6738 raft_consensus.cc:482] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:03.759202 6738 raft_consensus.cc:504] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:56:03.759866 6738 leader_election.cc:288] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:03.761957 22054 leader_election.cc:334] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:03.762207 22054 leader_election.cc:334] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:03.762331 22054 leader_election.cc:302] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:03.762650 6738 raft_consensus.cc:2726] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:03.763578 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:03.766264 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:03.769510 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:03.769623 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:03.818964 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:03.915004 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:03.918910 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:03.967231 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:03.972913 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:04.016081 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:04.030726 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.031421 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.038408 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.078501 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:04.084065 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.089848 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:04.110787 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:04.117563 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.132719 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:04.164153 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.207075 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.224716 21923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 01:56:04.245827 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.251788 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.253268 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.317793 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.323330 22185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 01:56:04.324450 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.332249 21791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 01:56:04.333807 24537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 191 similar messages]
W20240502 01:56:04.367004 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.372303 22054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240502 01:56:04.399945 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.416013 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:04.417029 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:04.435279 24934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 01:56:04.442278 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:04.514060 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:04.518316 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.555302 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.560542 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.644155 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.704478 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.715255 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:04.731001 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:04.738621 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:04.741937 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:04.806380 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:04.808547 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:04.827698 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:04.842087 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:04.843642 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:04.870014 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:04.870224 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:04.908772 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:04.923496 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:04.929922 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:04.933354 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:04.951967 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:56:04.955302 6758 raft_consensus.cc:482] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:04.955497 6758 raft_consensus.cc:504] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:04.956286 6758 leader_election.cc:288] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:04.957988 21791 leader_election.cc:334] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:04.958213 21791 leader_election.cc:334] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:04.958317 21791 leader_election.cc:302] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:04.958551 6758 raft_consensus.cc:2726] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:04.962837 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:04.999260 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:05.005393 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:05.015714 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:56:05.018819 6759 raft_consensus.cc:482] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:05.019060 6759 raft_consensus.cc:504] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:05.019719 6759 leader_election.cc:288] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:05.021348 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:05.021533 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:05.021631 21791 leader_election.cc:302] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:05.021872 6759 raft_consensus.cc:2726] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:05.059165 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:05.060950 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:05.074965 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:05.099346 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:05.143456 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:05.189209 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:05.211728 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:05.216996 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:05.270583 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:05.292184 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:05.301610 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:05.322338 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:05.365674 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:05.374640 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:05.424229 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:05.466569 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:05.469199 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:05.479151 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:05.481357 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:05.492041 6747 raft_consensus.cc:482] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:05.492226 6747 raft_consensus.cc:504] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:05.492821 6747 leader_election.cc:288] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:05.494289 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:05.494501 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:05.494628 24537 leader_election.cc:302] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:05.494894 6747 raft_consensus.cc:2726] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:05.498596 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:05.588842 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:56:05.640640 6754 raft_consensus.cc:482] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:05.640929 6754 raft_consensus.cc:504] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:05.641713 6754 leader_election.cc:288] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:05.643570 21923 leader_election.cc:334] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:05.643770 21923 leader_election.cc:334] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:05.643862 21923 leader_election.cc:302] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06543b05a16f42099bc86db2a0284412; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:05.644068 6754 raft_consensus.cc:2726] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:56:05.646947 6749 raft_consensus.cc:482] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:05.647109 6749 raft_consensus.cc:504] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:56:05.647747 6749 leader_election.cc:288] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:56:05.649441 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:05.649609 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:05.649719 22054 leader_election.cc:302] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:05.650015 6749 raft_consensus.cc:2726] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:05.702497 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:05.719666 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:05.725019 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:05.808240 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:05.838577 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:05.928529 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:06.007115 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.062778 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:06.120155 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:06.198032 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.216583 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.243227 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:06.246929 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.277689 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.287648 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:06.332628 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.398001 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:06.399488 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.408196 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:06.446149 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:56:06.447742 6653 raft_consensus.cc:482] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:06.447961 6653 raft_consensus.cc:504] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:56:06.448634 6653 leader_election.cc:288] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:06.450076 22185 leader_election.cc:334] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:06.450264 22185 leader_election.cc:334] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:06.450342 22185 leader_election.cc:302] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:06.450578 6653 raft_consensus.cc:2726] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:06.457476 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:06.485641 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:06.515586 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.522608 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:06.541844 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.561517 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:56:06.587452 6653 raft_consensus.cc:482] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:06.587751 6653 raft_consensus.cc:504] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:06.588595 6653 leader_election.cc:288] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:06.593667 22185 leader_election.cc:334] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:06.593901 22185 leader_election.cc:334] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:06.594029 22185 leader_election.cc:302] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:06.594348 6653 raft_consensus.cc:2726] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:06.619500 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.631839 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.643332 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:06.678912 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.690519 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.703140 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.721195 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.734766 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:56:06.734951 6759 raft_consensus.cc:482] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:06.735162 6759 raft_consensus.cc:504] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:06.735970 6759 leader_election.cc:288] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:56:06.738318 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:06.738557 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:06.738693 21791 leader_election.cc:302] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:06.738970 6759 raft_consensus.cc:2726] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:06.817528 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.830014 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.872294 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.878901 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.890872 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.894950 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:06.910315 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.925169 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:06.927481 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.931859 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:06.958182 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.028527 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:07.029691 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:07.071857 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.075831 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.175215 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:07.189098 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:07.195295 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.209622 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:07.271536 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.298517 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:07.303103 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.318666 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.364459 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.420755 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.428345 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.432981 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:07.435870 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:56:07.439033 6758 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:56:07.439240 6758 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
W20240502 01:56:07.439742 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:07.440762 22248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:56:07.441038 22248 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
I20240502 01:56:07.440294 6758 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
W20240502 01:56:07.441378 21791 leader_election.cc:334] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:07.442200 21791 leader_election.cc:341] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 36d358202b544e74bcc378060082df4d
I20240502 01:56:07.442580 21791 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:07.442888 6772 raft_consensus.cc:2726] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:07.467778 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:07.477720 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.511340 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:07.525420 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.552371 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:07.561147 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.594703 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.608902 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.614447 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.660338 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.660965 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.670730 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:56:07.723421 6749 raft_consensus.cc:482] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:07.723621 6749 raft_consensus.cc:504] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:07.724301 6749 leader_election.cc:288] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:07.725553 22054 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:07.725723 22054 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:07.725816 22054 leader_election.cc:302] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:07.726084 6749 raft_consensus.cc:2726] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:07.734036 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.773646 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.796976 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:07.797806 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.853579 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:07.888461 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.939361 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.944355 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.969276 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.973958 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:07.982095 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 01:56:08.024407 6774 raft_consensus.cc:482] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:08.024619 6774 raft_consensus.cc:504] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:56:08.025396 6774 leader_election.cc:288] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:08.026921 22054 leader_election.cc:334] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:08.027140 22054 leader_election.cc:334] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:08.027228 22054 leader_election.cc:302] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:08.027472 6774 raft_consensus.cc:2726] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:08.061110 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:08.076450 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:08.077545 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:08.119486 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:08.253876 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:08.256597 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:08.286906 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:08.336987 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 01:56:08.393203 6774 raft_consensus.cc:482] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:08.393438 6774 raft_consensus.cc:504] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:56:08.394119 6774 leader_election.cc:288] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:56:08.395874 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:08.396081 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:08.396199 22054 leader_election.cc:302] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:08.396461 6774 raft_consensus.cc:2726] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:08.427152 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:08.476488 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:08.481320 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:08.545922 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:08.606032 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:08.649340 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 01:56:08.694836 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling FlushMRSOp(4b924192d8da4a8bb7df2e6c5e4cbd4f): perf score=0.033370
W20240502 01:56:08.715986 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:08.762120 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:56:08.782189 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: FlushMRSOp(4b924192d8da4a8bb7df2e6c5e4cbd4f) complete. Timing: real 0.087s user 0.067s sys 0.014s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":438,"dirs.queue_time_us":981,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1249,"drs_written":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4,"lbm_write_time_us":892,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1435,"threads_started":3,"wal-append.queue_time_us":377}
I20240502 01:56:08.785777 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling UndoDeltaBlockGCOp(4b924192d8da4a8bb7df2e6c5e4cbd4f): 4921 bytes on disk
I20240502 01:56:08.786814 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: UndoDeltaBlockGCOp(4b924192d8da4a8bb7df2e6c5e4cbd4f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
W20240502 01:56:08.819085 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:08.823462 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:08.857736 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:08.864091 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:08.871003 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:08.957513 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:08.979895 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:09.016660 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.043809 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:09.057447 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.066454 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.068920 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.072755 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.076027 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.082180 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.111208 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.115567 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.161458 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:09.168510 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.188819 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:09.220713 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:09.225188 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.233528 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:09.246090 21923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240502 01:56:09.278095 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.322060 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.323457 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.332509 21791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 01:56:09.348738 22185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240502 01:56:09.349711 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 01:56:09.361171 6758 raft_consensus.cc:482] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:09.361402 6758 raft_consensus.cc:504] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:09.362226 6758 leader_election.cc:288] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:09.364248 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:09.364480 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:09.364607 21791 leader_election.cc:302] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
W20240502 01:56:09.365235 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 01:56:09.365289 6758 raft_consensus.cc:2726] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:09.402724 24537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 193 similar messages]
W20240502 01:56:09.403630 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.430701 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:09.431248 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:09.441612 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.447340 22054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 01:56:09.473538 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:09.483639 24934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240502 01:56:09.519785 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.527783 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:09.559001 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:09.568256 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:09.651360 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.716364 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:09.731081 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.733959 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 01:56:09.737807 24649 maintenance_manager.cc:419] P 0c64d40b5d7644bd8b29feaa80eab72f: Scheduling FlushMRSOp(3cf315f0117444ae9c5390a1e19460d9): perf score=0.033402
W20240502 01:56:09.738193 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:09.768842 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 01:56:09.796730 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling FlushMRSOp(ad3cd6fff977431b9cd5df6d083bef80): perf score=0.033352
I20240502 01:56:09.830876 24540 maintenance_manager.cc:643] P 0c64d40b5d7644bd8b29feaa80eab72f: FlushMRSOp(3cf315f0117444ae9c5390a1e19460d9) complete. Timing: real 0.092s user 0.060s sys 0.021s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":415,"dirs.queue_time_us":417,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1589,"drs_written":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4,"lbm_write_time_us":955,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":11335,"threads_started":3,"wal-append.queue_time_us":10889}
I20240502 01:56:09.834576 24649 maintenance_manager.cc:419] P 0c64d40b5d7644bd8b29feaa80eab72f: Scheduling UndoDeltaBlockGCOp(3cf315f0117444ae9c5390a1e19460d9): 4834 bytes on disk
I20240502 01:56:09.835662 24540 maintenance_manager.cc:643] P 0c64d40b5d7644bd8b29feaa80eab72f: UndoDeltaBlockGCOp(3cf315f0117444ae9c5390a1e19460d9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
W20240502 01:56:09.846009 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:09.861282 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: FlushMRSOp(ad3cd6fff977431b9cd5df6d083bef80) complete. Timing: real 0.064s user 0.058s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":369,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1899,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1498,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":530,"threads_started":2,"wal-append.queue_time_us":424}
I20240502 01:56:09.863823 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling UndoDeltaBlockGCOp(ad3cd6fff977431b9cd5df6d083bef80): 4581 bytes on disk
I20240502 01:56:09.864360 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: UndoDeltaBlockGCOp(ad3cd6fff977431b9cd5df6d083bef80) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240502 01:56:09.865972 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling FlushMRSOp(daaadb773202454894890c3379c4a58d): perf score=0.033346
W20240502 01:56:09.881345 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:09.881412 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:09.887941 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:09.904628 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:09.921449 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:09.925585 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:09.926635 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: FlushMRSOp(daaadb773202454894890c3379c4a58d) complete. Timing: real 0.060s user 0.038s sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":64,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1832,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":859,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":13404,"threads_started":1,"wal-append.queue_time_us":13516}
I20240502 01:56:09.929205 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling UndoDeltaBlockGCOp(daaadb773202454894890c3379c4a58d): 4834 bytes on disk
I20240502 01:56:09.933442 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: UndoDeltaBlockGCOp(daaadb773202454894890c3379c4a58d) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":3073,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
W20240502 01:56:09.955819 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:09.985527 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:10.019270 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:10.023442 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:10.026674 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:10.067384 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:10.071642 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:10.127739 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:10.130992 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:10.137364 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:10.159843 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:10.173018 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:10.183537 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:10.193189 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:10.259325 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:10.269543 6774 raft_consensus.cc:482] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:10.269801 6774 raft_consensus.cc:504] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:56:10.270682 6774 leader_election.cc:288] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:56:10.272725 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:10.272958 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:10.273073 22054 leader_election.cc:302] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:10.273340 6774 raft_consensus.cc:2726] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:10.329568 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:10.354988 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:10.357838 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:10.397305 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:10.446707 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:10.449194 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:10.472610 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:10.484220 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:10.499357 6778 raft_consensus.cc:482] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:10.499552 6778 raft_consensus.cc:504] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:10.500216 6778 leader_election.cc:288] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:10.502786 21923 leader_election.cc:334] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:10.503161 21923 leader_election.cc:334] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:10.503300 21923 leader_election.cc:302] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06543b05a16f42099bc86db2a0284412; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:10.503605 6778 raft_consensus.cc:2726] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:10.519295 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:10.540313 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:10.542429 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:10.694504 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:10.739085 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:10.755882 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:10.764345 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:10.774375 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:10.844815 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:10.966850 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:10.993279 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:11.081389 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:11.168007 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:11.176669 6772 raft_consensus.cc:482] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:11.176863 6772 raft_consensus.cc:504] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:11.177505 6772 leader_election.cc:288] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:11.179105 21791 leader_election.cc:334] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:11.179301 21791 leader_election.cc:334] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:11.179416 21791 leader_election.cc:302] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:11.179859 6772 raft_consensus.cc:2726] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:56:11.196180 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling FlushMRSOp(fedbec8860904a2cb5294037b7a4bcad): perf score=0.033372
W20240502 01:56:11.229516 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 01:56:11.236346 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: FlushMRSOp(fedbec8860904a2cb5294037b7a4bcad) complete. Timing: real 0.040s user 0.028s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":594,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1781,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":853,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":715,"threads_started":2,"wal-append.queue_time_us":298}
I20240502 01:56:11.238409 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling UndoDeltaBlockGCOp(fedbec8860904a2cb5294037b7a4bcad): 4834 bytes on disk
I20240502 01:56:11.238934 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: UndoDeltaBlockGCOp(fedbec8860904a2cb5294037b7a4bcad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240502 01:56:11.322486 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:11.347972 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:56:11.350138 24649 maintenance_manager.cc:419] P 0c64d40b5d7644bd8b29feaa80eab72f: Scheduling FlushMRSOp(ec67f45b2b604c129d8572c5ca6acbc4): perf score=0.033381
W20240502 01:56:11.379313 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:11.380185 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:56:11.390764 24540 maintenance_manager.cc:643] P 0c64d40b5d7644bd8b29feaa80eab72f: FlushMRSOp(ec67f45b2b604c129d8572c5ca6acbc4) complete. Timing: real 0.040s user 0.024s sys 0.014s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":431,"dirs.run_cpu_time_us":255,"dirs.run_wall_time_us":1752,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":741,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":595,"threads_started":2,"wal-append.queue_time_us":465}
I20240502 01:56:11.393111 24649 maintenance_manager.cc:419] P 0c64d40b5d7644bd8b29feaa80eab72f: Scheduling UndoDeltaBlockGCOp(ec67f45b2b604c129d8572c5ca6acbc4): 4921 bytes on disk
I20240502 01:56:11.393688 24540 maintenance_manager.cc:643] P 0c64d40b5d7644bd8b29feaa80eab72f: UndoDeltaBlockGCOp(ec67f45b2b604c129d8572c5ca6acbc4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
W20240502 01:56:11.414841 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:11.422297 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:11.426296 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:11.438799 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:11.446051 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:11.469924 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:11.482168 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:11.482146 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:11.505023 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:11.535847 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 01:56:11.539144 6771 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:56:11.539367 6771 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:56:11.540396 6771 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:56:11.540663 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:56:11.540952 21854 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
W20240502 01:56:11.541568 22185 leader_election.cc:341] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 36d358202b544e74bcc378060082df4d
W20240502 01:56:11.541764 22185 leader_election.cc:334] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:11.541860 22185 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:11.542124 6771 raft_consensus.cc:2726] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:11.554178 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:11.554872 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:11.566462 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:11.569425 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:11.588701 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:11.640745 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:11.641780 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:11.650290 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:11.670063 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:11.684448 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:11.720957 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:11.732494 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:11.774092 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:56:11.807327 6802 raft_consensus.cc:482] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:11.807523 6802 raft_consensus.cc:504] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:11.808182 6802 leader_election.cc:288] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:11.809685 22054 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:11.809882 22054 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:11.809980 22054 leader_election.cc:302] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:11.810242 6802 raft_consensus.cc:2726] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:11.861434 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:11.863611 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:11.877786 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:11.878790 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:11.947290 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:11.951298 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:11.973357 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:11.976280 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.006304 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.023931 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.032982 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:12.066336 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:12.091598 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:12.118805 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.124415 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:12.135161 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:56:12.151747 24649 maintenance_manager.cc:419] P 0c64d40b5d7644bd8b29feaa80eab72f: Scheduling FlushMRSOp(9bceab60366b4ade98f86072d391e7f4): perf score=0.033388
W20240502 01:56:12.152360 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:12.183719 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:56:12.191587 24540 maintenance_manager.cc:643] P 0c64d40b5d7644bd8b29feaa80eab72f: FlushMRSOp(9bceab60366b4ade98f86072d391e7f4) complete. Timing: real 0.040s user 0.032s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":370,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1440,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":716,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":535,"threads_started":2,"wal-append.queue_time_us":373}
I20240502 01:56:12.194012 24649 maintenance_manager.cc:419] P 0c64d40b5d7644bd8b29feaa80eab72f: Scheduling UndoDeltaBlockGCOp(9bceab60366b4ade98f86072d391e7f4): 4834 bytes on disk
I20240502 01:56:12.194571 24540 maintenance_manager.cc:643] P 0c64d40b5d7644bd8b29feaa80eab72f: UndoDeltaBlockGCOp(9bceab60366b4ade98f86072d391e7f4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
W20240502 01:56:12.253552 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:56:12.290299 6772 raft_consensus.cc:482] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:12.290509 6772 raft_consensus.cc:504] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:12.291307 6772 leader_election.cc:288] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:56:12.293829 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:12.294023 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:12.294128 21791 leader_election.cc:302] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:12.294687 6772 raft_consensus.cc:2726] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:12.315089 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.327303 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.390237 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.400614 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:56:12.404660 6820 raft_consensus.cc:482] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:12.404875 6820 raft_consensus.cc:504] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:56:12.405565 6820 leader_election.cc:288] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:12.407562 22185 leader_election.cc:334] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:12.407766 22185 leader_election.cc:334] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:12.407867 22185 leader_election.cc:302] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:12.408238 6820 raft_consensus.cc:2726] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:12.410667 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:12.457269 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:12.476119 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:12.481153 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.485883 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.489734 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:12.567356 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.570852 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.580840 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.583182 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.608215 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:12.667006 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.671330 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.685619 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.710865 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.716176 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.753643 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.760561 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.826706 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.896062 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:56:12.907725 6774 raft_consensus.cc:482] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:12.907920 6774 raft_consensus.cc:504] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:56:12.908753 6774 leader_election.cc:288] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:12.910741 22054 leader_election.cc:334] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:12.910969 22054 leader_election.cc:334] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:12.911084 22054 leader_election.cc:302] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:12.911350 6774 raft_consensus.cc:2726] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:12.978682 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:12.991185 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:13.007091 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:13.024976 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:13.036334 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:13.051800 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:56:13.070935 6771 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:56:13.071157 6771 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:56:13.071892 6771 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:56:13.072436 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:56:13.072675 21857 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
W20240502 01:56:13.072994 22185 leader_election.cc:341] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 36d358202b544e74bcc378060082df4d
W20240502 01:56:13.073782 22185 leader_election.cc:334] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:13.073917 22185 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:13.074172 6771 raft_consensus.cc:2726] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:13.096944 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:13.097743 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:13.149351 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:13.233119 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:13.234220 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:13.271378 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:13.335487 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:56:13.367928 6820 raft_consensus.cc:482] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:13.368121 6820 raft_consensus.cc:504] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:13.368732 6820 leader_election.cc:288] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:13.370604 22185 leader_election.cc:334] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:13.370802 22185 leader_election.cc:334] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:13.370901 22185 leader_election.cc:302] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:13.371265 6820 raft_consensus.cc:2726] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:13.474824 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:13.478497 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:13.485982 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:13.520181 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:13.536662 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:13.636791 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:13.648381 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:13.709942 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:13.785745 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:13.798502 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:13.841661 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:13.847424 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:13.859616 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:13.871433 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:13.874004 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:13.876168 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:13.927409 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:13.980546 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:13.989774 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:13.995944 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:14.022573 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:14.089445 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:14.095459 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:14.102718 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:14.107192 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:14.125993 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:14.150748 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:14.157809 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:14.185212 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 01:56:14.199414 6813 raft_consensus.cc:482] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:14.199616 6813 raft_consensus.cc:504] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:14.200261 6813 leader_election.cc:288] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:14.202539 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:14.202765 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:14.202857 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 01:56:14.202875 24537 leader_election.cc:302] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:14.203150 6813 raft_consensus.cc:2726] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:14.204356 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:14.221695 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:14.236723 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:14.282809 21923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 01:56:14.289003 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:14.292330 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:14.319589 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:14.343621 21791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 85 similar messages]
W20240502 01:56:14.347783 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:14.356344 22185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240502 01:56:14.413731 24537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 195 similar messages]
W20240502 01:56:14.414718 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:14.441393 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:14.454679 22054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240502 01:56:14.482412 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:14.517578 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:14.519676 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:14.526206 24934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 01:56:14.569859 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:14.583674 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:14.597215 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:14.622406 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:14.629817 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:14.669606 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:14.697901 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:14.735468 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:14.735826 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:14.770721 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:14.823415 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:14.834255 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:14.933326 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.005828 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.012776 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:15.020711 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.026449 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:15.042460 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.062081 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.062121 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.065594 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:15.082190 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.102340 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:15.103215 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.153496 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.167660 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.190933 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.211400 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.230616 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.286131 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:56:15.308573 6829 raft_consensus.cc:482] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:15.308799 6829 raft_consensus.cc:504] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
W20240502 01:56:15.309165 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:56:15.309638 6829 leader_election.cc:288] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:56:15.313050 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:15.313968 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:15.314069 21791 leader_election.cc:302] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:15.314345 6829 raft_consensus.cc:2726] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:15.420826 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.454017 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.484133 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.509399 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.514623 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:15.519874 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.530082 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:15.575642 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:15.609750 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:15.647114 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:15.651585 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:15.668354 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:15.682845 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:15.828861 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:15.908246 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.912166 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:15.944636 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:15.992712 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:16.064441 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:16.069267 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:16.136303 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:16.189343 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:16.202281 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:16.224141 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.241108 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.267927 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.344681 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:16.350857 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.356963 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:16.369050 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.414203 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:16.436657 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.441725 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.457981 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:16.465179 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.487056 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:16.508152 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.526803 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.551285 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.576467 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:16.618361 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:16.624214 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:16.630111 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.633643 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:16.665443 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.683790 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.698107 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.714907 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:16.741528 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:16.768088 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.789417 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.812405 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:16.840865 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:16.849464 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.917965 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.938987 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:16.941294 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:16.995273 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:17.014241 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:17.114897 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:17.140296 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:17.140602 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:17.145488 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:17.227461 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:17.267845 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:17.278345 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:17.289701 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:17.307299 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:17.332648 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:17.343335 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:17.365309 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:17.384171 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:17.412195 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:17.445152 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:17.463604 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:17.492271 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:17.497764 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:17.552983 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:17.571316 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:56:17.579488 6839 raft_consensus.cc:482] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:17.579735 6839 raft_consensus.cc:504] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
W20240502 01:56:17.580163 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:56:17.580377 6839 leader_election.cc:288] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:17.581441 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:17.582183 22054 leader_election.cc:334] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:17.582764 22054 leader_election.cc:334] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:17.582935 22054 leader_election.cc:302] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:17.583197 6839 raft_consensus.cc:2726] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:17.610270 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:17.623708 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:56:17.624053 6843 raft_consensus.cc:482] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:17.624205 6843 raft_consensus.cc:504] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:17.624734 6843 leader_election.cc:288] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:17.626089 21791 leader_election.cc:334] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:17.626351 21791 leader_election.cc:334] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:17.626436 21791 leader_election.cc:302] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:17.626721 6843 raft_consensus.cc:2726] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:17.640437 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:17.650919 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:17.663111 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:17.769323 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:17.774700 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:17.830029 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:17.868690 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:17.894308 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:56:17.906999 6820 raft_consensus.cc:482] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:17.907243 6820 raft_consensus.cc:504] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:56:17.907967 6820 leader_election.cc:288] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:17.909850 22185 leader_election.cc:334] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:17.910063 22185 leader_election.cc:334] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:17.910185 22185 leader_election.cc:302] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:17.910466 6820 raft_consensus.cc:2726] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:17.911700 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:17.973623 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:18.020004 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:18.080017 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:18.091321 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:18.099179 6840 raft_consensus.cc:482] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:18.099375 6840 raft_consensus.cc:504] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:18.100162 6840 leader_election.cc:288] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:18.102077 21923 leader_election.cc:334] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:18.102241 21923 leader_election.cc:334] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:18.102317 21923 leader_election.cc:302] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06543b05a16f42099bc86db2a0284412; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:18.102581 6840 raft_consensus.cc:2726] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:18.125584 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:18.141971 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:18.143877 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:18.159272 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:18.278725 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:18.355810 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:18.373685 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:18.392041 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:18.401307 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:18.448148 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:18.456986 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:18.463497 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:18.634537 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:18.650511 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:18.733013 6820 raft_consensus.cc:482] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:18.733240 6820 raft_consensus.cc:504] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:18.733882 6820 leader_election.cc:288] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:18.735411 22185 leader_election.cc:334] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:18.735612 22185 leader_election.cc:334] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:18.735693 22185 leader_election.cc:302] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:18.735996 6820 raft_consensus.cc:2726] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:18.746352 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:18.748260 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:18.763586 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:18.775710 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:18.877050 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:18.892956 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:18.910212 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:18.921229 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:18.931864 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:18.938344 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:18.958771 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:18.962399 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:18.983431 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:18.986573 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.006011 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.010192 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:19.046749 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:19.082252 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:19.100553 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.110278 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.131549 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:56:19.147063 6847 raft_consensus.cc:482] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:19.147272 6847 raft_consensus.cc:504] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:19.148100 6847 leader_election.cc:288] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:19.150815 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:19.151008 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:19.151098 24537 leader_election.cc:302] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:19.151309 6847 raft_consensus.cc:2726] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:19.171356 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:19.210613 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:19.224956 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.231045 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.232331 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:19.281191 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.307258 21923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240502 01:56:19.308322 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.315698 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.329001 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:19.355214 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:19.357678 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:19.374342 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:19.392177 21791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 84 similar messages]
W20240502 01:56:19.393070 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.394266 22185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240502 01:56:19.427956 24537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 192 similar messages]
W20240502 01:56:19.476392 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.506076 22054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240502 01:56:19.521605 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.525969 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.563334 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:19.576076 24934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240502 01:56:19.593940 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:19.596374 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.599865 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.650359 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:19.653275 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:19.669965 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:19.710052 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.727312 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.759547 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.783562 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.804297 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:19.813490 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:19.815162 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.881718 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:19.884240 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:56:19.896137 6834 raft_consensus.cc:482] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:19.896291 6834 raft_consensus.cc:504] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:19.896898 6834 leader_election.cc:288] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:19.898756 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:19.898911 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:19.898980 21791 leader_election.cc:302] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:19.899202 6834 raft_consensus.cc:2726] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:19.934219 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:19.939266 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:19.962796 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:19.995486 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:20.009490 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:20.049911 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:20.066538 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:20.107522 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:20.111182 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:20.133184 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:20.141954 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:20.283843 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:20.297133 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:20.325951 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:20.344774 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:20.354635 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:20.378772 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:20.457142 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:20.497272 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:20.510413 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:20.524318 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:20.533547 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:20.534312 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:20.566481 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:20.599346 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:20.757696 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:20.759963 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:20.787918 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:20.801728 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:20.802492 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:20.904259 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:20.983877 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:21.003214 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:21.052470 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:21.081961 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:21.125419 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:21.218307 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:21.288219 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:21.288805 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:21.316272 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:21.409089 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:21.412860 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:21.427306 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:21.428273 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:21.449358 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:21.458755 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:21.473555 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:21.486985 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:21.489329 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:21.490897 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:21.499723 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:21.563159 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:21.567349 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:21.568372 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:21.585528 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:21.609719 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:21.622763 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:21.631806 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:21.745751 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:21.752511 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:21.752566 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:21.846148 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:21.875957 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:21.880357 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:21.908985 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:21.931854 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:21.948443 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:21.961684 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:21.978597 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:22.026891 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:22.065292 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:22.070767 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:22.131927 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:22.133746 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:22.150157 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:22.157863 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:22.159065 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:22.173626 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:22.232214 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:22.250891 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:22.251588 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:22.343941 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:22.346259 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:22.354357 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:22.395390 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:22.398181 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:22.412053 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:22.417032 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:22.457808 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:22.460388 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:22.541859 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:22.546737 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:22.580824 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:22.618918 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:22.673185 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:22.674104 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:56:22.687247 6868 raft_consensus.cc:482] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:22.687435 6868 raft_consensus.cc:504] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:22.688347 6868 leader_election.cc:288] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:56:22.689265 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:22.690656 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:22.690871 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:22.690950 21791 leader_election.cc:302] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:22.691188 6868 raft_consensus.cc:2726] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:22.707084 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:22.724351 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:22.725193 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:22.733240 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:22.743136 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:22.820780 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:22.837828 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:56:22.860612 6839 raft_consensus.cc:482] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:22.860852 6839 raft_consensus.cc:504] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:22.861629 6839 leader_election.cc:288] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:22.863377 22054 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:22.863621 22054 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:22.863708 22054 leader_election.cc:302] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:22.863946 6839 raft_consensus.cc:2726] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:22.884826 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:22.931919 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:22.958297 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:23.059175 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:23.067200 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:23.128454 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:23.144471 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:23.183918 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:23.196486 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:23.264245 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:23.277179 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:23.290699 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:23.298856 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:56:23.316718 6860 raft_consensus.cc:482] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:23.316998 6860 raft_consensus.cc:504] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:23.317749 6860 leader_election.cc:288] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:23.326658 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:23.327584 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:23.327718 21791 leader_election.cc:302] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:23.328155 6860 raft_consensus.cc:2726] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:23.356559 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:23.402390 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:23.525408 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:23.550720 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:23.551981 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:23.572818 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:23.725698 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:23.735471 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:23.817755 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:23.827901 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:23.849913 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:23.863140 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:23.863994 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:23.921552 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:23.932412 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:23.939652 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:23.954339 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:23.983994 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:23.988567 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:23.991714 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:24.007302 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:24.011246 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:24.079689 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:24.107203 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:24.140687 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:24.157872 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:24.160204 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:24.188803 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:24.203622 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:24.248070 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:24.250946 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:24.289678 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:24.330124 21923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 01:56:24.331755 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:24.344727 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:24.366413 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:24.377005 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:24.383117 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:24.404683 22185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240502 01:56:24.417307 21791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 01:56:24.426054 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:24.430191 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:24.442811 24537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 191 similar messages]
W20240502 01:56:24.445456 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:24.467607 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:24.484231 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:24.504482 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:24.510354 22054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240502 01:56:24.511173 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:24.557572 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:24.575790 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:24.582223 24934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240502 01:56:24.601431 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:24.612871 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:24.613379 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:24.699885 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:24.715731 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:24.763043 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:24.788520 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:24.803792 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:24.850005 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:24.881655 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:24.883666 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:24.889823 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:24.893090 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:24.918763 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:24.919517 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:24.979486 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:25.027035 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:25.038808 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:25.042713 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:25.061617 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:25.123756 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:25.140861 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:25.143460 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:25.168462 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:25.171233 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:25.184763 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:25.263159 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:25.267549 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 01:56:25.271693 6870 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:56:25.271910 6870 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:56:25.273393 6870 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
W20240502 01:56:25.273875 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:25.274439 22185 leader_election.cc:341] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 36d358202b544e74bcc378060082df4d
W20240502 01:56:25.275480 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 01:56:25.276121 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:56:25.276409 21854 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
W20240502 01:56:25.276646 22185 leader_election.cc:334] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:25.276767 22185 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:25.277020 6870 raft_consensus.cc:2726] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:25.309388 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:25.368288 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:25.441018 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:25.441818 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:25.442217 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:25.444209 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:25.519871 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:25.520999 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:25.571537 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:25.582005 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 01:56:25.634270 6874 raft_consensus.cc:482] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:25.634442 6874 raft_consensus.cc:504] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:56:25.635048 6874 leader_election.cc:288] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:25.636621 22054 leader_election.cc:334] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:25.636859 22054 leader_election.cc:334] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:25.637408 22054 leader_election.cc:302] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:25.637718 6874 raft_consensus.cc:2726] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:25.709220 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:25.778332 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:25.845289 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:25.861361 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:25.896762 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:25.906697 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:26.054419 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.103916 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:26.120564 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:26.159781 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:26.194803 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.246961 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:26.279230 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.279357 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.294800 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:26.335644 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:26.394685 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:26.423892 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.443464 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:26.448982 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.471632 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.485724 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:26.493882 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:26.503839 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:26.509073 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:26.566807 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.584651 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.595436 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.612772 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.640448 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.704044 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:26.727916 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.747231 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.750043 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.771605 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.781685 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:26.796039 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.811399 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:26.835695 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.854449 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:26.908903 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.923202 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:26.937011 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.951583 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:26.982678 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:26.984519 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:26.986100 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:26.991716 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:26.993990 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:27.002038 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.043725 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:27.074347 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.093144 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.179581 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.202840 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:27.207211 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.213632 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:27.227907 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.270403 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.281982 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.304230 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 01:56:27.342554 6883 raft_consensus.cc:482] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:27.342757 6883 raft_consensus.cc:504] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:56:27.343529 6883 leader_election.cc:288] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:56:27.345063 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:27.345227 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:27.345300 22054 leader_election.cc:302] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:27.345525 6883 raft_consensus.cc:2726] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:27.352238 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.371507 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.386859 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:27.387708 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.439213 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.455231 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.473492 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.490679 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.507674 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.538184 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:27.541570 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:27.583572 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.741235 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:27.760568 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.786880 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.795684 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:27.844708 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:27.855278 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:27.857061 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.909983 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:27.923328 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:27.950286 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:27.984118 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:27.985776 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:28.031170 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:28.088090 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:28.094308 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:28.143410 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:28.155072 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:28.285549 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:28.326140 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:28.405179 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:28.549278 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:28.551074 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:28.596836 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:28.652758 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:28.691931 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:28.712110 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:28.761096 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:28.761624 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:28.767752 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:28.795791 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:28.799073 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:28.802642 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:28.852581 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:28.906661 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:28.972486 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:28.974514 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.000437 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:29.012818 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:29.020004 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:29.052398 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:29.078439 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.105262 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:29.126752 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.151535 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.183367 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:56:29.189358 6886 raft_consensus.cc:482] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:29.189548 6886 raft_consensus.cc:504] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:29.190215 6886 leader_election.cc:288] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:29.191771 21791 leader_election.cc:334] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:29.191951 21791 leader_election.cc:334] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:29.192039 21791 leader_election.cc:302] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:29.192332 6886 raft_consensus.cc:2726] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:29.194160 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.231287 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.237368 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.240963 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.241007 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:29.271279 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.280122 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:29.333582 21923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 01:56:29.341615 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.343282 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.394557 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:29.398310 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.399327 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:29.418030 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:29.440948 21791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240502 01:56:29.453608 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.460804 22185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240502 01:56:29.461606 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.482164 24537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 186 similar messages]
W20240502 01:56:29.483143 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:29.536058 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:29.537187 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.547086 22054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240502 01:56:29.609197 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.612731 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.613989 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:29.624702 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:29.625445 24934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 01:56:29.632462 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:29.645659 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:29.655493 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:29.670805 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:29.695430 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:29.703091 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:29.724787 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:29.735878 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.746091 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:29.773432 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:56:29.775161 6898 raft_consensus.cc:482] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:29.775331 6898 raft_consensus.cc:504] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:29.775954 6898 leader_election.cc:288] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:29.777842 21923 leader_election.cc:334] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:29.778026 21923 leader_election.cc:334] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:29.778117 21923 leader_election.cc:302] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06543b05a16f42099bc86db2a0284412; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:29.778414 6898 raft_consensus.cc:2726] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:29.781860 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.794683 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:29.821543 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:29.835343 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.840315 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:29.865237 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:29.877925 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:29.950773 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:30.050894 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:30.146221 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:30.152832 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:30.199709 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:30.286022 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:30.302446 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:30.317510 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:30.335614 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:30.344237 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:30.353948 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:30.362080 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:30.395653 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:30.537487 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:30.581281 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:30.596529 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:30.604683 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:30.626807 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:30.628365 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:30.653827 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:30.681206 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:30.705060 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:30.722568 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:30.724228 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:30.790143 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:30.801046 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:56:30.829838 6820 raft_consensus.cc:482] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:30.830063 6820 raft_consensus.cc:504] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:56:30.830736 6820 leader_election.cc:288] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:30.838737 22185 leader_election.cc:334] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:30.838943 22185 leader_election.cc:334] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:30.839013 22185 leader_election.cc:302] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:30.839236 6820 raft_consensus.cc:2726] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:30.886734 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:30.926920 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:30.970216 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:31.145558 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.193368 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.195399 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:31.278542 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:31.295360 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:31.304299 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:31.304442 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.327620 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.367149 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.405277 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.456470 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.487056 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.492807 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:31.494099 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:31.566160 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.581367 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:31.667515 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:31.695034 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:31.702211 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.704053 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:31.718711 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.741462 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.748114 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.756489 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:31.819388 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.828706 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:31.829438 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.842689 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.852193 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:31.903543 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.903878 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.922422 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:31.924264 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:31.929893 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.942297 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:31.956701 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:31.958362 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:31.960980 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:31.972127 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:32.000149 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:32.000149 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:32.033792 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:32.087824 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:32.103137 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:32.109437 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:32.115988 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:32.125311 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:32.152594 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:32.169970 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:32.178886 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 01:56:32.186803 6883 raft_consensus.cc:482] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
W20240502 01:56:32.186883 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:32.187026 6883 raft_consensus.cc:504] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:32.187672 6883 leader_election.cc:288] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:32.189363 22054 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:32.189559 22054 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:32.189674 22054 leader_election.cc:302] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:32.189981 6883 raft_consensus.cc:2726] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:32.234663 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:32.294713 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:32.335287 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:32.337052 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:32.369958 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:32.409984 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:32.413012 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:32.435829 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:32.489840 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:32.632309 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:32.635841 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:32.640995 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:32.667407 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:32.716621 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:32.732213 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:32.791126 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:32.805115 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:32.810894 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:32.825048 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:32.827226 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:32.889688 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:32.931067 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:32.947389 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:33.003427 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:33.016258 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:33.067791 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:33.099864 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:33.142513 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:33.149968 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:33.152338 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:33.243480 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:33.255028 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:33.284070 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:33.332828 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:33.355335 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:33.358572 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:33.407196 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:33.441514 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:33.472815 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:33.543992 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 01:56:33.563360 6907 raft_consensus.cc:482] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:33.563596 6907 raft_consensus.cc:504] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:56:33.564245 6907 leader_election.cc:288] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:56:33.566016 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:33.566252 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:33.566362 22054 leader_election.cc:302] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:33.566643 6907 raft_consensus.cc:2726] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:33.570981 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:33.584124 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:33.612237 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:33.635872 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:33.656157 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:33.793186 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:33.806334 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:33.809943 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:33.882884 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:33.892448 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:33.954129 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:34.013906 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:34.017388 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:34.128777 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:34.130698 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:34.135643 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:34.195160 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:34.204107 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:34.228243 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:34.232335 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:34.238163 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:34.250823 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:34.260103 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:34.292089 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:34.302707 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:34.316849 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:34.346282 21923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240502 01:56:34.347350 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:34.393579 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:34.402942 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:34.413583 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:34.414906 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:34.437582 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:34.460803 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:34.468796 22185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 01:56:34.481638 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:34.505924 21791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 01:56:34.524924 24537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 192 similar messages]
W20240502 01:56:34.525314 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:34.526286 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 01:56:34.559343 6892 raft_consensus.cc:482] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:34.559546 6892 raft_consensus.cc:504] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:34.560168 6892 leader_election.cc:288] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:34.561880 22185 leader_election.cc:334] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:34.562199 22185 leader_election.cc:334] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:34.562294 22185 leader_election.cc:302] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:34.562574 6820 raft_consensus.cc:2726] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:34.563098 22054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240502 01:56:34.564098 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:34.565409 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:34.577369 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:34.590847 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:34.609496 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:34.610460 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:34.667429 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:34.687538 24934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240502 01:56:34.693410 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:34.696197 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:34.704370 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:34.725746 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:34.739337 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:34.740841 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:34.783886 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:34.794951 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:34.824499 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:34.831270 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:34.858063 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:34.866218 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:34.901655 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:34.911950 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:34.924052 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:34.991065 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:35.037679 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:35.042435 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:35.055681 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:35.097470 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:35.180397 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:35.184278 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:35.184376 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:35.197548 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:35.300302 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:35.305147 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:35.349690 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:35.364828 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:35.429133 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:35.454663 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:35.492578 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:35.496426 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:35.511794 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:35.518983 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:35.620731 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:35.707374 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:35.730333 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:35.805852 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:35.834115 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:35.873320 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:35.902546 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:35.907721 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:35.909256 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:35.942654 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:35.975104 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:35.976418 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:36.149864 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:36.153252 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:36.170636 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:36.215708 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 01:56:36.241113 6927 raft_consensus.cc:482] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:36.241312 6927 raft_consensus.cc:504] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:36.241989 6927 leader_election.cc:288] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:36.243717 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:36.244324 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:36.244493 24537 leader_election.cc:302] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:36.244757 6927 raft_consensus.cc:2726] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:36.263186 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:36.304746 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:36.329687 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:36.352015 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:36.359722 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:36.414996 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:36.425573 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:36.426489 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:36.455520 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 01:56:36.458328 6934 raft_consensus.cc:482] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:36.458552 6934 raft_consensus.cc:504] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
W20240502 01:56:36.458974 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 01:56:36.459313 6934 leader_election.cc:288] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:36.460983 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:36.461185 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:36.461275 21791 leader_election.cc:302] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:36.461509 6934 raft_consensus.cc:2726] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:36.468731 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:36.470397 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:36.497153 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:56:36.515269 6892 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:56:36.515480 6892 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:56:36.517284 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:56:36.517552 21856 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
I20240502 01:56:36.516875 6892 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
W20240502 01:56:36.519073 22185 leader_election.cc:334] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:36.519423 22185 leader_election.cc:341] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 36d358202b544e74bcc378060082df4d
I20240502 01:56:36.519560 22185 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:36.519982 6892 raft_consensus.cc:2726] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:36.521508 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:36.650816 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:36.653487 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:36.703775 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:36.728289 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:36.748620 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:36.754915 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:36.798825 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:36.899996 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:36.937700 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 01:56:36.949826 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling FlushMRSOp(0d246b118c1c4388ac04b3775ddbee89): perf score=0.033374
W20240502 01:56:36.954254 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:36.967619 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:36.990262 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: FlushMRSOp(0d246b118c1c4388ac04b3775ddbee89) complete. Timing: real 0.040s user 0.021s sys 0.017s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":317,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1266,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":993,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":482,"threads_started":2,"wal-append.queue_time_us":364}
I20240502 01:56:36.992144 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling UndoDeltaBlockGCOp(0d246b118c1c4388ac04b3775ddbee89): 4921 bytes on disk
I20240502 01:56:36.992692 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: UndoDeltaBlockGCOp(0d246b118c1c4388ac04b3775ddbee89) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
W20240502 01:56:36.997336 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:37.004909 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:37.040570 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:56:37.057276 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:37.061894 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:37.084427 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:37.089913 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:37.107282 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:37.121343 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:37.142637 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:37.157274 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:37.199749 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:37.201632 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:37.230648 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:37.243691 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:37.244251 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:56:37.246484 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling FlushMRSOp(dcc8171c991b48ddbfda8007e3a87a38): perf score=0.033388
W20240502 01:56:37.259809 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:37.262790 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:37.268177 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:56:37.283354 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: FlushMRSOp(dcc8171c991b48ddbfda8007e3a87a38) complete. Timing: real 0.037s user 0.027s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1415,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":769,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":203,"threads_started":1,"wal-append.queue_time_us":317}
I20240502 01:56:37.285786 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling FlushMRSOp(3964f5f080f145a5923dac2f076637c8): perf score=0.033340
W20240502 01:56:37.328038 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:37.332722 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:37.336159 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:37.350499 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 01:56:37.357268 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: FlushMRSOp(3964f5f080f145a5923dac2f076637c8) complete. Timing: real 0.071s user 0.050s sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1334,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":808,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7475,"threads_started":1,"wal-append.queue_time_us":7584}
I20240502 01:56:37.359578 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling UndoDeltaBlockGCOp(dcc8171c991b48ddbfda8007e3a87a38): 4918 bytes on disk
I20240502 01:56:37.360481 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: UndoDeltaBlockGCOp(dcc8171c991b48ddbfda8007e3a87a38) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240502 01:56:37.362861 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling UndoDeltaBlockGCOp(3964f5f080f145a5923dac2f076637c8): 4581 bytes on disk
I20240502 01:56:37.363598 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: UndoDeltaBlockGCOp(3964f5f080f145a5923dac2f076637c8) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
W20240502 01:56:37.381726 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:37.393430 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:37.398113 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:37.409907 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:37.431334 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:37.455454 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:37.461902 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:37.467478 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:37.497730 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:37.553930 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:37.599496 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:37.648021 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:37.686791 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:37.723219 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:37.729754 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:37.745903 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:37.793484 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:37.820847 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:37.822872 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:37.847990 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:37.936491 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:37.946216 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:38.001477 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:38.039007 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:38.052639 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:38.061394 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:38.106914 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:38.146854 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:38.181243 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:38.349609 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:56:38.373467 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling FlushMRSOp(d4b6dfe0e8da4352b136d6f97d473bad): perf score=0.033372
W20240502 01:56:38.404733 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:56:38.408998 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: FlushMRSOp(d4b6dfe0e8da4352b136d6f97d473bad) complete. Timing: real 0.035s user 0.030s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":368,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1419,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":594,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":704,"threads_started":2,"wal-append.queue_time_us":561}
W20240502 01:56:38.409410 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:38.410753 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:56:38.411650 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling UndoDeltaBlockGCOp(d4b6dfe0e8da4352b136d6f97d473bad): 4581 bytes on disk
I20240502 01:56:38.412137 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: UndoDeltaBlockGCOp(d4b6dfe0e8da4352b136d6f97d473bad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
W20240502 01:56:38.432178 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:38.467945 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:38.503309 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:38.519630 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:38.542691 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:38.573562 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:38.581125 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:38.589602 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:38.589836 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:56:38.644459 24649 maintenance_manager.cc:419] P 0c64d40b5d7644bd8b29feaa80eab72f: Scheduling FlushMRSOp(84d602fa045e4cfa9b698d3e92e5cf8f): perf score=0.033389
I20240502 01:56:38.710558 24540 maintenance_manager.cc:643] P 0c64d40b5d7644bd8b29feaa80eab72f: FlushMRSOp(84d602fa045e4cfa9b698d3e92e5cf8f) complete. Timing: real 0.066s user 0.049s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":337,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":1435,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":736,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":449,"threads_started":2,"wal-append.queue_time_us":303}
I20240502 01:56:38.712714 24649 maintenance_manager.cc:419] P 0c64d40b5d7644bd8b29feaa80eab72f: Scheduling UndoDeltaBlockGCOp(84d602fa045e4cfa9b698d3e92e5cf8f): 4921 bytes on disk
I20240502 01:56:38.713323 24540 maintenance_manager.cc:643] P 0c64d40b5d7644bd8b29feaa80eab72f: UndoDeltaBlockGCOp(84d602fa045e4cfa9b698d3e92e5cf8f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
W20240502 01:56:38.775435 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:38.789553 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:38.809921 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:38.831434 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:38.839864 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:38.914639 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:38.958782 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:38.990128 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:38.990439 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:39.030650 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:39.053288 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:39.062340 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:39.070554 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:39.099704 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:39.134215 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:39.213639 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:39.239344 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:39.254209 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:39.261073 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:39.265204 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:39.291815 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:39.322877 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:39.347832 21923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 140 similar messages]
I20240502 01:56:39.421964 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling FlushMRSOp(da66e78c2c5044b2924682be0b643b0a): perf score=0.033401
W20240502 01:56:39.423182 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:39.437007 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:56:39.458719 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: FlushMRSOp(da66e78c2c5044b2924682be0b643b0a) complete. Timing: real 0.036s user 0.030s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":342,"dirs.run_cpu_time_us":267,"dirs.run_wall_time_us":2337,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":807,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":454,"threads_started":2,"wal-append.queue_time_us":307}
I20240502 01:56:39.461473 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling UndoDeltaBlockGCOp(da66e78c2c5044b2924682be0b643b0a): 4582 bytes on disk
W20240502 01:56:39.461539 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:56:39.462046 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: UndoDeltaBlockGCOp(da66e78c2c5044b2924682be0b643b0a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
W20240502 01:56:39.483879 22185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240502 01:56:39.527702 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:39.539361 21791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 01:56:39.540313 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:39.542142 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:56:39.544651 6966 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:56:39.544804 6966 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:56:39.545730 6966 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:56:39.546255 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:56:39.546542 22250 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
W20240502 01:56:39.546967 21791 leader_election.cc:334] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:39.547281 21791 leader_election.cc:341] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 36d358202b544e74bcc378060082df4d
I20240502 01:56:39.547394 21791 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:39.547684 6966 raft_consensus.cc:2726] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:39.553263 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:39.562628 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:39.583472 22054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240502 01:56:39.584308 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:39.601176 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:39.611364 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:56:39.619767 6940 raft_consensus.cc:482] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:39.619930 6940 raft_consensus.cc:504] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:39.620479 6940 leader_election.cc:288] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:39.621358 24537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 191 similar messages]
W20240502 01:56:39.622175 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:39.622311 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:39.622383 24537 leader_election.cc:302] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:39.622634 6940 raft_consensus.cc:2726] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:39.631250 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:39.640985 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:39.663991 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:39.665201 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:39.671097 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:39.678849 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:56:39.683753 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:39.690850 24934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 01:56:39.716084 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:39.721125 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:39.735848 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:39.736001 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:39.740865 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:39.757025 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:39.813735 6944 raft_consensus.cc:482] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:39.813968 6944 raft_consensus.cc:504] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:56:39.814659 6944 leader_election.cc:288] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:56:39.816218 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:39.816448 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:39.816545 22054 leader_election.cc:302] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:39.816830 6944 raft_consensus.cc:2726] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:39.825966 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:39.853109 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:39.867489 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:39.883373 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:39.892356 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:39.896749 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:39.923727 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:39.940116 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:39.949925 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:39.968786 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:40.044199 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:40.093717 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:40.162986 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:40.163937 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:40.230401 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:40.253479 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:40.263893 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:40.322393 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:40.332921 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:40.334807 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:40.411489 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:40.424471 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:40.431583 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:40.435346 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:40.542165 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:40.573714 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:40.576227 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:40.619958 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:40.651087 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:40.692521 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:40.731463 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:40.762542 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:40.767210 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:56:40.829597 6961 raft_consensus.cc:482] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
W20240502 01:56:40.829702 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 01:56:40.829823 6961 raft_consensus.cc:504] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:56:40.830521 6961 leader_election.cc:288] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:40.832365 22054 leader_election.cc:334] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:40.832542 22054 leader_election.cc:334] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:40.832640 22054 leader_election.cc:302] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:40.832942 6961 raft_consensus.cc:2726] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:40.846028 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:40.875203 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:40.891108 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:40.916347 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:40.989830 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:40.992856 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:41.019594 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:41.072685 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:41.074867 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:41.126435 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:41.177086 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:41.208243 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:41.279510 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:41.317126 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:41.319046 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:41.355078 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:41.473349 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:41.548594 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:41.578473 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:41.584151 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:41.631886 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:41.643313 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:41.692474 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:41.695668 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:41.699687 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:41.710307 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:41.728926 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:41.730929 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:41.766034 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:41.810377 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:41.835341 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:41.841841 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:56:41.842101 6938 raft_consensus.cc:482] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:41.842386 6938 raft_consensus.cc:504] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:41.844321 6938 leader_election.cc:288] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:41.847365 21923 leader_election.cc:334] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:41.847684 21923 leader_election.cc:334] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:41.847800 21923 leader_election.cc:302] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06543b05a16f42099bc86db2a0284412; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:41.848090 6938 raft_consensus.cc:2726] T 61f537605bc746a0807e3c0a03ff2cc3 P 06543b05a16f42099bc86db2a0284412 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:41.848201 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:41.857522 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:41.866523 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:41.943616 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:41.948596 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:41.988762 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:41.996964 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:42.049038 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.058809 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.059554 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:56:42.060716 6969 raft_consensus.cc:482] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:42.060917 6969 raft_consensus.cc:504] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:42.061635 6969 leader_election.cc:288] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:56:42.063030 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:42.063222 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:42.063320 21791 leader_election.cc:302] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:42.063601 6969 raft_consensus.cc:2726] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:42.075906 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:42.093200 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:56:42.130179 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.130461 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:42.137322 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.158166 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:42.162974 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.172909 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:42.199450 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.233095 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:42.266607 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.290678 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.296154 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.298800 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.303205 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:42.309016 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:42.326661 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:42.334308 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.346678 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:42.368809 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:42.370582 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.401777 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.422189 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:42.474238 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:42.515403 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:42.663691 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.703181 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:42.722390 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.762238 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:42.797770 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.816295 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.841795 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.842414 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.856175 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:42.904824 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:42.909125 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:42.940377 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:42.962252 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 01:56:43.001968 6961 raft_consensus.cc:482] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:43.002159 6961 raft_consensus.cc:504] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:43.002876 6961 leader_election.cc:288] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:43.004377 22054 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:43.004630 22054 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:43.004724 22054 leader_election.cc:302] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:43.005030 6961 raft_consensus.cc:2726] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:43.013859 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:43.024396 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:43.031688 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:43.039216 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:43.220642 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:43.253752 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:43.304131 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:43.315354 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:43.326457 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:43.348196 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:43.373725 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:43.401288 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:43.421631 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:43.437022 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:43.476560 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:43.489357 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:43.492134 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:43.526742 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:43.611574 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:43.628160 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:43.669377 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:43.694947 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:43.811057 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:43.933331 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:43.975453 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:43.979517 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:44.002437 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:44.007014 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:44.026026 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:44.032060 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:44.192107 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:44.220093 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:44.224714 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:44.227883 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:44.241319 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:44.284015 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:44.298663 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:44.299597 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:44.311838 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:44.361557 21923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 01:56:44.361696 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:44.365430 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:44.367930 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:44.374356 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:44.385058 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:44.399222 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:44.421079 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:44.429633 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:44.453553 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:44.471716 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:44.487388 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:44.491246 22185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 01:56:44.523288 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:44.548060 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:44.548666 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:44.587416 22054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240502 01:56:44.591029 21791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 86 similar messages]
W20240502 01:56:44.592008 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:44.609181 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:44.660225 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:56:44.691967 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:44.696620 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:44.701962 24934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 01:56:44.706096 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:44.730513 24537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 195 similar messages]
W20240502 01:56:44.737267 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:44.742635 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:44.843901 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:44.851838 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:44.862397 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:44.864923 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:44.906706 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:44.923846 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:44.927951 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:44.942451 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:44.957418 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:44.958945 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:44.985198 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:44.995645 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:45.012707 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 01:56:45.026232 6987 raft_consensus.cc:482] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:45.026401 6987 raft_consensus.cc:504] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:45.027024 6987 leader_election.cc:288] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:45.028506 21791 leader_election.cc:334] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:45.028710 21791 leader_election.cc:334] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:45.028807 21791 leader_election.cc:302] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:45.029109 6987 raft_consensus.cc:2726] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:45.035282 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:45.039076 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:45.096527 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:45.150547 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:45.158149 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:45.259131 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:45.306344 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:45.344705 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:45.367945 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:45.388772 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:45.391091 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:45.438078 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:45.466272 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:45.476240 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:45.481894 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:45.494356 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:45.507962 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:45.546324 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:45.587625 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:45.636253 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:45.648332 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:45.745671 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:45.747071 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:45.861294 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:45.894686 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:45.900931 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:45.911249 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:45.936030 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:45.939121 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:45.950955 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:45.979827 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:45.983471 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:46.036022 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:46.132858 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:46.147727 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:46.154560 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:46.167618 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:46.284754 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:46.358016 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:46.395244 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:46.444286 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:46.503762 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:46.544183 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:46.547246 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:46.553095 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:46.581786 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:46.603428 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:46.604223 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:46.604840 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:46.620205 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:46.640328 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:46.647862 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:46.663475 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:46.715884 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:46.730803 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:46.744402 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:46.762462 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:46.783108 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:46.797478 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:46.825860 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:46.839192 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:46.944857 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:46.970788 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:46.981781 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:46.997186 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:47.027226 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:47.057474 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:47.080749 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:47.103037 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:47.123509 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:47.145962 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:47.172173 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:56:47.174633 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:47.190659 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:47.192554 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:47.252810 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:47.263200 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:47.266726 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:47.269232 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:47.304004 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:47.346233 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:47.356215 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:47.365016 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:47.381951 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:56:47.384865 7000 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:56:47.385126 7000 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:56:47.386008 7000 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:56:47.386467 22250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "759257ea7bef415ca698e60d82199d77" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
I20240502 01:56:47.386705 22250 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759257ea7bef415ca698e60d82199d77 in term 1.
W20240502 01:56:47.388252 21791 leader_election.cc:334] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:47.388531 21791 leader_election.cc:341] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 36d358202b544e74bcc378060082df4d
I20240502 01:56:47.388659 21791 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:47.389015 7000 raft_consensus.cc:2726] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:47.434772 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:47.468751 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:47.471648 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:47.515919 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:47.531888 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:47.589259 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:47.609724 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:47.712019 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:47.723711 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:47.786451 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:47.835788 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:47.839668 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:47.865760 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:56:47.875100 7010 raft_consensus.cc:482] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:47.875340 7010 raft_consensus.cc:504] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:56:47.876147 7010 leader_election.cc:288] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:56:47.885362 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:47.885617 22054 leader_election.cc:334] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:47.885771 22054 leader_election.cc:302] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:47.886054 7010 raft_consensus.cc:2726] T b60b948dae4841dd9c0919f2fcac5299 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:47.888284 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:47.900805 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:47.917315 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:47.925055 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:47.935119 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:47.948057 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:47.970899 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:48.024993 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:48.071907 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:48.157979 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:48.165562 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:48.193964 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:48.219620 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:48.222846 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:48.247015 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:48.262185 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:48.328014 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:48.356127 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:48.367911 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:48.368570 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:48.402696 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:48.520447 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:48.540266 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:48.541067 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:48.585448 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:48.592291 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:48.603127 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:48.676052 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:48.738472 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:48.764446 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:48.782687 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:48.800608 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:48.812520 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:48.870741 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:48.923290 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:48.965214 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:49.025655 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:49.044026 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:49.045706 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:49.055116 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:49.075512 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:49.080494 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 01:56:49.094282 7007 raft_consensus.cc:482] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:49.094447 7007 raft_consensus.cc:504] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:56:49.095175 7007 leader_election.cc:288] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:49.097229 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:49.097232 22185 leader_election.cc:334] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:49.097544 22185 leader_election.cc:334] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:49.097625 22185 leader_election.cc:302] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:49.097821 7007 raft_consensus.cc:2726] T 8b377c7402194b42a8eabc747f668fb2 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:49.099670 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:49.135634 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:49.189229 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:49.190246 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:49.232909 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:49.237473 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:49.252810 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:49.298544 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:49.300041 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:49.304997 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:49.308657 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:49.334100 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:49.353240 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:49.403772 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:49.406512 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:49.448690 21923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 146 similar messages]
W20240502 01:56:49.516384 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:49.552508 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:49.556823 22185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240502 01:56:49.572011 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:49.579296 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:49.612709 22054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240502 01:56:49.614871 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:49.616716 21791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240502 01:56:49.626019 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:49.629680 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:49.661284 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:56:49.691813 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:49.696516 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:49.709829 24934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 01:56:49.738947 24537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 193 similar messages]
W20240502 01:56:49.749096 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:49.821525 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:49.824407 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:49.844770 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:49.857105 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:49.881434 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:49.924342 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:49.956346 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:50.008724 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:50.015005 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:50.034322 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:50.036047 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:50.096351 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:50.121215 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:50.147970 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:50.154590 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:50.195391 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:50.207854 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:50.220624 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:50.230855 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:50.266670 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:50.268769 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:50.319314 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:50.380853 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:50.384588 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:50.439822 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:50.447130 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:50.489318 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:50.495054 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:50.509461 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:50.606292 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:50.636464 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:50.663414 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:50.718353 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:50.728160 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:50.740100 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:50.801591 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:50.812381 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:50.817162 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:50.821151 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:50.860355 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:50.884039 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:50.893635 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:50.945976 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:50.952839 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 01:56:51.000365 7029 raft_consensus.cc:482] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:51.000603 7029 raft_consensus.cc:504] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:51.001350 7029 leader_election.cc:288] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:56:51.003288 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:51.003511 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:51.003643 21791 leader_election.cc:302] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:51.003878 7029 raft_consensus.cc:2726] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:51.025588 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:51.035961 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:51.095955 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:51.148566 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:51.155176 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:51.223137 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:56:51.235466 7019 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:56:51.235656 7019 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:56:51.236552 7019 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:56:51.237112 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:56:51.237355 21856 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
W20240502 01:56:51.237849 22185 leader_election.cc:341] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 36d358202b544e74bcc378060082df4d
W20240502 01:56:51.238003 22185 leader_election.cc:334] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:51.238091 22185 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:51.238315 7019 raft_consensus.cc:2726] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:51.285329 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:51.295404 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:51.362627 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:51.385095 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:51.527742 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:51.536923 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:51.538403 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:51.541136 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:51.558053 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:51.558971 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:51.604142 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:51.623234 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:51.626166 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:51.650380 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:51.670488 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:51.765200 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:51.768373 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:51.773850 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:51.809370 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:51.823237 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:51.843129 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:51.849792 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:51.864933 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:51.911443 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:51.913353 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:51.949710 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:51.950129 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:52.018221 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:52.032928 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:52.049984 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.056906 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:52.063196 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:56:52.065791 7029 raft_consensus.cc:482] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:52.065966 7029 raft_consensus.cc:504] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:52.066535 7029 leader_election.cc:288] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:52.068535 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:52.068744 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:52.068856 21791 leader_election.cc:302] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:52.069180 7029 raft_consensus.cc:2726] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:52.092978 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:52.105051 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:52.189594 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.201931 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.205890 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:52.222875 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:52.224601 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.229568 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.243839 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:52.264218 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.305753 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.314344 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:52.407555 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:52.423449 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.433132 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.439177 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:56:52.487627 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.536049 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:52.538053 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:52.539182 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:52.564801 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:52.627774 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:52.642943 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.670796 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.698904 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.735626 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:52.748306 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.756444 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.775306 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.791636 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:52.809186 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.833981 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.879030 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:52.942761 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:52.946945 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.949836 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:52.991928 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:53.092772 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:53.118415 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:53.143893 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:53.179579 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:53.310312 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:53.362663 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:53.370182 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:53.410203 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:53.411192 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:53.440706 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:53.471714 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:53.473167 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:53.497787 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:53.527938 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:53.532961 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:53.563989 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:53.576472 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:53.592000 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:56:53.642191 7030 raft_consensus.cc:482] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:53.642392 7030 raft_consensus.cc:504] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:53.643168 7030 leader_election.cc:288] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:56:53.645017 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:53.645243 21791 leader_election.cc:334] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:53.645366 21791 leader_election.cc:302] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:53.645671 7030 raft_consensus.cc:2726] T a2c3475bcad94e3dad4be87376e500bc P 759257ea7bef415ca698e60d82199d77 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:53.649358 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:53.667788 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:53.695050 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:53.737382 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:53.796219 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:53.858530 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:53.961138 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:53.968492 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:53.988505 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.037900 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.041464 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.098383 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.150712 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.217519 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.228291 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:54.243834 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:54.246168 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:56:54.264346 7042 raft_consensus.cc:482] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:54.264547 7042 raft_consensus.cc:504] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:54.265192 7042 leader_election.cc:288] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:54.266674 22185 leader_election.cc:334] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:54.266866 22185 leader_election.cc:334] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:54.266966 22185 leader_election.cc:302] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2a28baa7ee2469294d94b7fe5455cff; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:54.267231 7042 raft_consensus.cc:2726] T 45ccb387e833453a950b1ef5b5282d29 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:54.291985 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:54.294294 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.347879 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:54.354200 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.360504 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.360970 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:54.396867 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.470587 21923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240502 01:56:54.471653 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:54.494951 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:54.509891 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.515592 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:54.526353 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:54.571520 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:54.578280 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.581835 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:54.598258 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:54.608855 22185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240502 01:56:54.647810 22054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240502 01:56:54.673806 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:54.683588 21791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111) [suppressed 84 similar messages]
W20240502 01:56:54.684437 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:54.689440 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.713218 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:56:54.719686 24934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 01:56:54.735146 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.744143 24537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 188 similar messages]
W20240502 01:56:54.745677 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:54.749946 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:54.758404 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.761768 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:54.791930 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.801646 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.803207 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.820590 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:54.838588 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.849184 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:54.855947 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:54.895277 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:54.929333 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.954741 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:54.957201 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:56:55.058952 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:56:55.096181 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:55.115617 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:55.127002 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:55.127923 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:55.128327 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:55.170416 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:55.198236 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:55.292516 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:55.332780 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:55.333351 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:55.333813 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:55.347537 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:55.371920 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:55.495409 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:55.501411 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:55.540733 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:55.542933 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:55.572041 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:55.623123 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:55.673110 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:55.683010 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:55.773667 21791 consensus_peers.cc:485] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:55.780191 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:55.792781 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:55.825682 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:55.828428 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:55.919999 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 01:56:55.962759 7048 raft_consensus.cc:482] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:55.962965 7048 raft_consensus.cc:504] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:56:55.963794 7048 leader_election.cc:288] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:55.965452 22054 leader_election.cc:334] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:55.965672 22054 leader_election.cc:334] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:55.965759 22054 leader_election.cc:302] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:55.965976 7048 raft_consensus.cc:2726] T 9ad2f9c27d284098a8e4f20ebc4b3d8c P c64840657cdf4b93980bd34120c94d9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:56.012792 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:56.015765 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:56.104333 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:56.130448 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:56.140498 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:56.152611 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:56.174439 21923 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:56.188740 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:56.199777 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:56.203687 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:56.248586 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:56.256202 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:56.289301 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:56.339957 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:56.421519 21923 consensus_peers.cc:485] T 0f8200b044ef44e1a0451baf2b1485d2 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:56.422237 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:56.505877 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:56.539436 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:56.559619 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:56.601483 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:56:56.629390 7047 raft_consensus.cc:482] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:56.629590 7047 raft_consensus.cc:504] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:56.630280 7047 leader_election.cc:288] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:56.632174 21791 leader_election.cc:334] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:56.632364 21791 leader_election.cc:334] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:56.632450 21791 leader_election.cc:302] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:56.632671 7047 raft_consensus.cc:2726] T 97c9eb341ce1470db791324a882a0271 P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:56.638739 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:56.670284 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:56.679140 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 01:56:56.708204 7047 raft_consensus.cc:482] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:56.708400 7047 raft_consensus.cc:504] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:56:56.709127 7047 leader_election.cc:288] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:56.710973 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:56.711202 21791 leader_election.cc:334] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:56.711275 21791 leader_election.cc:302] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:56.711548 7047 raft_consensus.cc:2726] T 0a1b190978514cb49efb2b97cc97a65f P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:56.737200 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:56.749742 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:56.792117 22054 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:56.821127 22054 consensus_peers.cc:485] T daaadb773202454894890c3379c4a58d P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:56.845602 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:56.859825 22054 consensus_peers.cc:485] T dea30e2e1ef6473f880a001e57f27171 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:56.896286 22054 consensus_peers.cc:485] T 7ce01a3c9d9948ba8f153ad150cf3cb0 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:56.956282 21791 consensus_peers.cc:485] T d52f4b5c815f4188a09eb12ec5375361 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:56.962936 21791 consensus_peers.cc:485] T b1c6c622638444bd854616902b8d91ad P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:56.975661 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:57.031306 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:56:57.047837 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:57.063575 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:56:57.067044 21837 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 01:56:57.067884 21923 consensus_peers.cc:485] T ac80dc6d73db4352b9864392badc16c1 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:56:57.069029 22054 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:57.078466 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:57.109403 22054 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:57.115239 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:57.118881 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:56:57.119959 24583 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 01:56:57.123378 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:57.134312 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 01:56:57.143280 24981 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:56:57.148509 22229 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 01:56:57.160446 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 01:56:57.177978 22100 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 01:56:57.179162 21969 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 01:56:57.185390 21923 consensus_peers.cc:485] T 9f05784e69ff49c8bed9720812a067a3 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:57.185585 21923 consensus_peers.cc:485] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:57.194244 21791 consensus_peers.cc:485] T 3be66f4b90824a21a5f41915c9041970 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:57.199563 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:57.225741 21923 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:57.248299 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:57.293272 22054 consensus_peers.cc:485] T 0eed9ee06a494588a32b4c951d6d3a14 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 01:56:57.295228 7042 raft_consensus.cc:482] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7449bb1df214bf4a5f93e4631312f08)
I20240502 01:56:57.295423 7042 raft_consensus.cc:504] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:56:57.296272 7042 leader_election.cc:288] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677)
I20240502 01:56:57.297063 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d358202b544e74bcc378060082df4d" candidate_uuid: "c2a28baa7ee2469294d94b7fe5455cff" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "759257ea7bef415ca698e60d82199d77" is_pre_election: true
I20240502 01:56:57.297331 21856 raft_consensus.cc:2454] T 36d358202b544e74bcc378060082df4d P 759257ea7bef415ca698e60d82199d77 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a28baa7ee2469294d94b7fe5455cff in term 1.
W20240502 01:56:57.298125 22185 leader_election.cc:334] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:57.299316 22185 leader_election.cc:341] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Not found: Tablet not found: 36d358202b544e74bcc378060082df4d
I20240502 01:56:57.299481 22185 leader_election.cc:302] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff; no voters: 0c64d40b5d7644bd8b29feaa80eab72f, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:57.300175 7085 raft_consensus.cc:2726] T 36d358202b544e74bcc378060082df4d P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:57.319296 21791 consensus_peers.cc:485] T 5a16aff54e6f4975a97776c214890f57 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:57.345402 21923 consensus_peers.cc:485] T bddb4a31d81d44afaabc247fd1449767 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:57.348124 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:56:57.350461 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:57.439940 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:57.441339 22054 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:57.442044 22185 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:57.472627 21791 consensus_peers.cc:485] T ae04b73f8a574264a26a647400a44235 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:57.479172 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:57.519855 24537 consensus_peers.cc:485] T 564f981f7f7c4f2480c4fd1e5b5c1ad8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:56:57.572299 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:57.620141 22185 consensus_peers.cc:485] T 1803e35563e84c84b90cbf608ecf6a93 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:57.645145 21791 consensus_peers.cc:462] T 5b3f40a9f1c34644b787905c28a54b29 P 759257ea7bef415ca698e60d82199d77 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:57.728693 21791 consensus_peers.cc:485] T 766241df5ee24264a5cb7176e6c97b91 P 759257ea7bef415ca698e60d82199d77 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:57.749356 22054 consensus_peers.cc:485] T 9bceab60366b4ade98f86072d391e7f4 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:57.784765 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:57.817373 21923 consensus_peers.cc:485] T 96c5ae723e7a43a0b91a743b0c192a67 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:57.888320 24537 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-taskITWpJv/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 1
Maximum Replica Count | 41
Average Replica Count | 20.625000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 1
Maximum | 3
Average | 2.894737
stderr: W20240502 01:54:02.360149 25056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:02.360471 25056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:02.369330 25056 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240502 01:54:02.956164 25127 meta_cache.cc:1015] Not found: LookupRpc { table: 'rebalance_test_table_2', partition-key: (<start>), attempt: 1 } failed: table rebalance_test_table_2 was deleted: Table deleted at 2024-05-02 01:54:02 UTC
I20240502 01:54:04.507017 25056 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 01:54:06.470520 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 906b29d0bc414f019de158cfa649177d: could not find any suitable replica to move from server 06543b05a16f42099bc86db2a0284412 to server 28157bc3e678428fb5f93c077d528513
W20240502 01:54:06.470818 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 17fb52b34dea4f27811e830b6350d802: could not find any suitable replica to move from server 06543b05a16f42099bc86db2a0284412 to server d7449bb1df214bf4a5f93e4631312f08
W20240502 01:54:06.471315 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 17fb52b34dea4f27811e830b6350d802: could not find any suitable replica to move from server c2a28baa7ee2469294d94b7fe5455cff to server 41c70f373c9d49a49891b03a12c6cb2b
W20240502 01:54:06.471530 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 906b29d0bc414f019de158cfa649177d: could not find any suitable replica to move from server 759257ea7bef415ca698e60d82199d77 to server d7449bb1df214bf4a5f93e4631312f08
W20240502 01:54:06.471730 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 596028c83e784a669c0b5819444e54be: could not find any suitable replica to move from server 759257ea7bef415ca698e60d82199d77 to server 28157bc3e678428fb5f93c077d528513
W20240502 01:54:06.471827 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 17fb52b34dea4f27811e830b6350d802: could not find any suitable replica to move from server 06543b05a16f42099bc86db2a0284412 to server 0c64d40b5d7644bd8b29feaa80eab72f
W20240502 01:54:06.471885 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 868965c86d0d413fbbeaca41748259c6: could not find any suitable replica to move from server c2a28baa7ee2469294d94b7fe5455cff to server 41c70f373c9d49a49891b03a12c6cb2b
W20240502 01:54:06.471956 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eaf29e0201fc4b738b1b30c13a773ffd: could not find any suitable replica to move from server c64840657cdf4b93980bd34120c94d9a to server d7449bb1df214bf4a5f93e4631312f08
W20240502 01:54:06.472038 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b462b431b51b4d45baf0ff5a495aa90a: could not find any suitable replica to move from server c2a28baa7ee2469294d94b7fe5455cff to server 0c64d40b5d7644bd8b29feaa80eab72f
W20240502 01:54:06.472153 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e67a4afb37114119a9d514153a1563bb: could not find any suitable replica to move from server 06543b05a16f42099bc86db2a0284412 to server d7449bb1df214bf4a5f93e4631312f08
W20240502 01:54:06.472204 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 906b29d0bc414f019de158cfa649177d: could not find any suitable replica to move from server 06543b05a16f42099bc86db2a0284412 to server 0c64d40b5d7644bd8b29feaa80eab72f
W20240502 01:54:06.472348 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table efd69cb962724c4f85c2825debc9eba0: could not find any suitable replica to move from server c2a28baa7ee2469294d94b7fe5455cff to server 0c64d40b5d7644bd8b29feaa80eab72f
W20240502 01:54:06.472409 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 596028c83e784a669c0b5819444e54be: could not find any suitable replica to move from server c64840657cdf4b93980bd34120c94d9a to server 0c64d40b5d7644bd8b29feaa80eab72f
W20240502 01:54:06.472478 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 714c5d94b5b64cb29dc398713b1727fe: could not find any suitable replica to move from server 759257ea7bef415ca698e60d82199d77 to server 28157bc3e678428fb5f93c077d528513
W20240502 01:54:06.472549 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7a17c6db79004eb696e7dd281fdc4f3b: could not find any suitable replica to move from server 759257ea7bef415ca698e60d82199d77 to server 41c70f373c9d49a49891b03a12c6cb2b
W20240502 01:54:06.472609 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 17fb52b34dea4f27811e830b6350d802: could not find any suitable replica to move from server c2a28baa7ee2469294d94b7fe5455cff to server 28157bc3e678428fb5f93c077d528513
W20240502 01:54:06.472721 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 27c9bf24f886426b92b80b01e4a8d669: could not find any suitable replica to move from server 06543b05a16f42099bc86db2a0284412 to server 0c64d40b5d7644bd8b29feaa80eab72f
W20240502 01:54:06.472838 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 906b29d0bc414f019de158cfa649177d: could not find any suitable replica to move from server c64840657cdf4b93980bd34120c94d9a to server 41c70f373c9d49a49891b03a12c6cb2b
W20240502 01:54:06.472941 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9827c9aba79e4ba094c6e9c0e32f36f4: could not find any suitable replica to move from server 06543b05a16f42099bc86db2a0284412 to server 0c64d40b5d7644bd8b29feaa80eab72f
W20240502 01:54:06.473076 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cd297aa2796f428db6b815990620e2a4: could not find any suitable replica to move from server 06543b05a16f42099bc86db2a0284412 to server 41c70f373c9d49a49891b03a12c6cb2b
W20240502 01:54:06.473212 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 27064ffcfcc643d6ad082b6cd6ea21db: could not find any suitable replica to move from server 759257ea7bef415ca698e60d82199d77 to server d7449bb1df214bf4a5f93e4631312f08
W20240502 01:54:06.473332 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 855af8637536470bb3f01aeac50e07e8: could not find any suitable replica to move from server 759257ea7bef415ca698e60d82199d77 to server 28157bc3e678428fb5f93c077d528513
W20240502 01:54:06.473488 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7a17c6db79004eb696e7dd281fdc4f3b: could not find any suitable replica to move from server c2a28baa7ee2469294d94b7fe5455cff to server 0c64d40b5d7644bd8b29feaa80eab72f
W20240502 01:54:06.473611 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5ffb0e9de5844f82a2ae0ec1f30c97ef: could not find any suitable replica to move from server 759257ea7bef415ca698e60d82199d77 to server 0c64d40b5d7644bd8b29feaa80eab72f
W20240502 01:54:06.473724 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 714c5d94b5b64cb29dc398713b1727fe: could not find any suitable replica to move from server c64840657cdf4b93980bd34120c94d9a to server d7449bb1df214bf4a5f93e4631312f08
W20240502 01:54:06.473842 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eaf29e0201fc4b738b1b30c13a773ffd: could not find any suitable replica to move from server c2a28baa7ee2469294d94b7fe5455cff to server 28157bc3e678428fb5f93c077d528513
I20240502 01:54:06.550103 25056 rebalancer_tool.cc:1074] tablet 071ad932ce95499fac4d7af7292b42ff: 'c64840657cdf4b93980bd34120c94d9a' -> 'd7449bb1df214bf4a5f93e4631312f08' move scheduled
I20240502 01:54:06.553016 25056 rebalancer_tool.cc:1084] tablet 68e1096b01964772ac02755f1ae21c53: 'c2a28baa7ee2469294d94b7fe5455cff' -> '41c70f373c9d49a49891b03a12c6cb2b' move ignored: Not found: 68e1096b01964772ac02755f1ae21c53: tablet not found
I20240502 01:54:06.553159 25056 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:54:06.742911 25503 meta_cache.cc:788] Not found: LookupRpcById { tablet: '6f1686dbe93c42b597f32422b6e8ac75', attempt: 1 } failed
W20240502 01:54:07.489377 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e87707ee294e7c9eed52900510d340: could not find any suitable replica to move from server d7449bb1df214bf4a5f93e4631312f08 to server c64840657cdf4b93980bd34120c94d9a
W20240502 01:54:07.489993 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e67a4afb37114119a9d514153a1563bb: could not find any suitable replica to move from server 06543b05a16f42099bc86db2a0284412 to server d7449bb1df214bf4a5f93e4631312f08
W20240502 01:54:07.490132 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cd297aa2796f428db6b815990620e2a4: could not find any suitable replica to move from server 06543b05a16f42099bc86db2a0284412 to server 41c70f373c9d49a49891b03a12c6cb2b
W20240502 01:54:07.490270 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c1a2048ad81b49729677603cbe361089: could not find any suitable replica to move from server d7449bb1df214bf4a5f93e4631312f08 to server c2a28baa7ee2469294d94b7fe5455cff
W20240502 01:54:07.490442 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9827c9aba79e4ba094c6e9c0e32f36f4: could not find any suitable replica to move from server c2a28baa7ee2469294d94b7fe5455cff to server 41c70f373c9d49a49891b03a12c6cb2b
W20240502 01:54:07.490547 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 596028c83e784a669c0b5819444e54be: could not find any suitable replica to move from server 06543b05a16f42099bc86db2a0284412 to server 41c70f373c9d49a49891b03a12c6cb2b
W20240502 01:54:07.490684 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b462b431b51b4d45baf0ff5a495aa90a: could not find any suitable replica to move from server c64840657cdf4b93980bd34120c94d9a to server 0c64d40b5d7644bd8b29feaa80eab72f
W20240502 01:54:07.490800 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eaf29e0201fc4b738b1b30c13a773ffd: could not find any suitable replica to move from server 06543b05a16f42099bc86db2a0284412 to server d7449bb1df214bf4a5f93e4631312f08
W20240502 01:54:07.490924 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 868965c86d0d413fbbeaca41748259c6: could not find any suitable replica to move from server 06543b05a16f42099bc86db2a0284412 to server 41c70f373c9d49a49891b03a12c6cb2b
W20240502 01:54:07.491036 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 714c5d94b5b64cb29dc398713b1727fe: could not find any suitable replica to move from server 759257ea7bef415ca698e60d82199d77 to server d7449bb1df214bf4a5f93e4631312f08
W20240502 01:54:07.491216 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7a17c6db79004eb696e7dd281fdc4f3b: could not find any suitable replica to move from server c2a28baa7ee2469294d94b7fe5455cff to server 28157bc3e678428fb5f93c077d528513
W20240502 01:54:07.491325 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 06ae89337ff143b5a107c994cbecc25a: could not find any suitable replica to move from server 0c64d40b5d7644bd8b29feaa80eab72f to server c2a28baa7ee2469294d94b7fe5455cff
W20240502 01:54:07.491505 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 855af8637536470bb3f01aeac50e07e8: could not find any suitable replica to move from server c2a28baa7ee2469294d94b7fe5455cff to server 28157bc3e678428fb5f93c077d528513
W20240502 01:54:07.491675 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 27064ffcfcc643d6ad082b6cd6ea21db: could not find any suitable replica to move from server c64840657cdf4b93980bd34120c94d9a to server 28157bc3e678428fb5f93c077d528513
W20240502 01:54:07.491806 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 596028c83e784a669c0b5819444e54be: could not find any suitable replica to move from server 759257ea7bef415ca698e60d82199d77 to server d7449bb1df214bf4a5f93e4631312f08
W20240502 01:54:07.491886 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b462b431b51b4d45baf0ff5a495aa90a: could not find any suitable replica to move from server c2a28baa7ee2469294d94b7fe5455cff to server 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:54:07.595935 25056 rebalancer_tool.cc:1074] tablet 4b924192d8da4a8bb7df2e6c5e4cbd4f: '759257ea7bef415ca698e60d82199d77' -> '41c70f373c9d49a49891b03a12c6cb2b' move scheduled
I20240502 01:54:07.713552 25056 rebalancer_tool.cc:1074] tablet 166e717fb9624ae58adb6699790daf7a: 'c2a28baa7ee2469294d94b7fe5455cff' -> '28157bc3e678428fb5f93c077d528513' move scheduled
I20240502 01:54:07.716239 25056 rebalancer_tool.cc:1084] tablet 37e185badcf64f078e0df67837cfe054: '06543b05a16f42099bc86db2a0284412' -> '0c64d40b5d7644bd8b29feaa80eab72f' move ignored: Not found: 37e185badcf64f078e0df67837cfe054: tablet not found
I20240502 01:54:07.716354 25056 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:54:07.924707 25663 meta_cache.cc:1015] Not found: LookupRpc { table: 'rebalancer_dynamic_table_new_name', partition-key: (<start>), attempt: 1 } failed: table rebalancer_dynamic_table_new_name was deleted: Table deleted at 2024-05-02 01:54:07 UTC
W20240502 01:54:08.574087 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 714c5d94b5b64cb29dc398713b1727fe: could not find any suitable replica to move from server 759257ea7bef415ca698e60d82199d77 to server 0c64d40b5d7644bd8b29feaa80eab72f
W20240502 01:54:08.574285 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7a17c6db79004eb696e7dd281fdc4f3b: could not find any suitable replica to move from server c64840657cdf4b93980bd34120c94d9a to server 0c64d40b5d7644bd8b29feaa80eab72f
W20240502 01:54:08.574472 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 27064ffcfcc643d6ad082b6cd6ea21db: could not find any suitable replica to move from server 06543b05a16f42099bc86db2a0284412 to server 28157bc3e678428fb5f93c077d528513
W20240502 01:54:08.574586 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9827c9aba79e4ba094c6e9c0e32f36f4: could not find any suitable replica to move from server c64840657cdf4b93980bd34120c94d9a to server 0c64d40b5d7644bd8b29feaa80eab72f
W20240502 01:54:08.574769 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 855af8637536470bb3f01aeac50e07e8: could not find any suitable replica to move from server 759257ea7bef415ca698e60d82199d77 to server 0c64d40b5d7644bd8b29feaa80eab72f
W20240502 01:54:08.574862 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e67a4afb37114119a9d514153a1563bb: could not find any suitable replica to move from server 06543b05a16f42099bc86db2a0284412 to server d7449bb1df214bf4a5f93e4631312f08
W20240502 01:54:08.574997 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b462b431b51b4d45baf0ff5a495aa90a: could not find any suitable replica to move from server c2a28baa7ee2469294d94b7fe5455cff to server d7449bb1df214bf4a5f93e4631312f08
W20240502 01:54:08.575062 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eaf29e0201fc4b738b1b30c13a773ffd: could not find any suitable replica to move from server c64840657cdf4b93980bd34120c94d9a to server d7449bb1df214bf4a5f93e4631312f08
W20240502 01:54:08.575145 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 868965c86d0d413fbbeaca41748259c6: could not find any suitable replica to move from server 06543b05a16f42099bc86db2a0284412 to server 0c64d40b5d7644bd8b29feaa80eab72f
W20240502 01:54:08.575280 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cd297aa2796f428db6b815990620e2a4: could not find any suitable replica to move from server c2a28baa7ee2469294d94b7fe5455cff to server 41c70f373c9d49a49891b03a12c6cb2b
W20240502 01:54:08.575388 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7a17c6db79004eb696e7dd281fdc4f3b: could not find any suitable replica to move from server 759257ea7bef415ca698e60d82199d77 to server 28157bc3e678428fb5f93c077d528513
W20240502 01:54:08.575449 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9827c9aba79e4ba094c6e9c0e32f36f4: could not find any suitable replica to move from server 06543b05a16f42099bc86db2a0284412 to server 0c64d40b5d7644bd8b29feaa80eab72f
I20240502 01:54:08.653146 25056 rebalancer_tool.cc:1074] tablet 3cf315f0117444ae9c5390a1e19460d9: '06543b05a16f42099bc86db2a0284412' -> '0c64d40b5d7644bd8b29feaa80eab72f' move scheduled
I20240502 01:54:08.768074 25056 rebalancer_tool.cc:1074] tablet db96ccbe81564bb589b0de65977b3321: 'c64840657cdf4b93980bd34120c94d9a' -> 'd7449bb1df214bf4a5f93e4631312f08' move scheduled
I20240502 01:54:08.833441 25056 rebalancer_tool.cc:1074] tablet d52f4b5c815f4188a09eb12ec5375361: 'c2a28baa7ee2469294d94b7fe5455cff' -> '28157bc3e678428fb5f93c077d528513' move scheduled
I20240502 01:54:08.949002 25056 rebalancer_tool.cc:1074] tablet ad3cd6fff977431b9cd5df6d083bef80: '759257ea7bef415ca698e60d82199d77' -> '41c70f373c9d49a49891b03a12c6cb2b' move scheduled
I20240502 01:54:09.000257 25056 rebalancer_tool.cc:1074] tablet 992544bbea6547de96298fd32d74b232: 'd7449bb1df214bf4a5f93e4631312f08' -> '06543b05a16f42099bc86db2a0284412' move scheduled
I20240502 01:54:09.082418 25056 rebalancer_tool.cc:1074] tablet b60b948dae4841dd9c0919f2fcac5299: '0c64d40b5d7644bd8b29feaa80eab72f' -> 'c64840657cdf4b93980bd34120c94d9a' move scheduled
I20240502 01:54:09.130556 25056 rebalancer_tool.cc:1074] tablet daaadb773202454894890c3379c4a58d: 'c2a28baa7ee2469294d94b7fe5455cff' -> '41c70f373c9d49a49891b03a12c6cb2b' move scheduled
I20240502 01:54:09.226703 25056 rebalancer_tool.cc:1074] tablet 03fb607ef0d9469daca1d6f3c5259fb2: '28157bc3e678428fb5f93c077d528513' -> '759257ea7bef415ca698e60d82199d77' move scheduled
I20240502 01:54:09.757674 25056 rebalancer_tool.cc:1074] tablet 766241df5ee24264a5cb7176e6c97b91: '06543b05a16f42099bc86db2a0284412' -> '28157bc3e678428fb5f93c077d528513' move scheduled
I20240502 01:54:10.038748 25056 rebalancer_tool.cc:1074] tablet e00feee2662a48d8a1a57f9e52275154: '0c64d40b5d7644bd8b29feaa80eab72f' -> 'c64840657cdf4b93980bd34120c94d9a' move scheduled
I20240502 01:54:10.127566 25056 rebalancer_tool.cc:1074] tablet d4b6dfe0e8da4352b136d6f97d473bad: 'c2a28baa7ee2469294d94b7fe5455cff' -> 'd7449bb1df214bf4a5f93e4631312f08' move scheduled
I20240502 01:54:10.206844 25056 rebalancer_tool.cc:1074] tablet fedbec8860904a2cb5294037b7a4bcad: '759257ea7bef415ca698e60d82199d77' -> '41c70f373c9d49a49891b03a12c6cb2b' move scheduled
I20240502 01:54:10.382963 25056 rebalancer_tool.cc:1074] tablet 0eed9ee06a494588a32b4c951d6d3a14: '06543b05a16f42099bc86db2a0284412' -> '28157bc3e678428fb5f93c077d528513' move scheduled
I20240502 01:54:10.713961 25056 rebalancer_tool.cc:1074] tablet ec67f45b2b604c129d8572c5ca6acbc4: 'c64840657cdf4b93980bd34120c94d9a' -> '0c64d40b5d7644bd8b29feaa80eab72f' move scheduled
I20240502 01:54:11.087172 25056 rebalancer_tool.cc:1074] tablet 0f8200b044ef44e1a0451baf2b1485d2: 'c2a28baa7ee2469294d94b7fe5455cff' -> 'd7449bb1df214bf4a5f93e4631312f08' move scheduled
I20240502 01:54:11.206305 25056 rebalancer_tool.cc:1074] tablet 9bceab60366b4ade98f86072d391e7f4: '759257ea7bef415ca698e60d82199d77' -> '0c64d40b5d7644bd8b29feaa80eab72f' move scheduled
E20240502 01:54:11.382987 26284 ksck_remote.cc:634] unable to open table rebalancer_dynamic_table_new_name: Not found: the table does not exist: table_name: "rebalancer_dynamic_table_new_name"
E20240502 01:54:18.376600 27141 ksck_remote.cc:634] unable to open table rebalancer_dynamic_table_new_name: Not found: the table does not exist: table_name: "rebalancer_dynamic_table_new_name"
W20240502 01:54:25.515486 27955 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.14.212.254:33157, user_credentials={real_user=slave}} blocked reactor thread for 68804.2us
E20240502 01:54:26.557597 28115 ksck_remote.cc:634] unable to open table rebalancer_dynamic_table: Not found: the table does not exist: table_name: "rebalancer_dynamic_table"
I20240502 01:54:29.136941 25056 rebalancer_tool.cc:1123] tablet 071ad932ce95499fac4d7af7292b42ff: 'c64840657cdf4b93980bd34120c94d9a' -> 'd7449bb1df214bf4a5f93e4631312f08' move is abandoned: Not found: 071ad932ce95499fac4d7af7292b42ff: tablet not found
W20240502 01:54:29.877091 28440 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.14.212.198:42375, user_credentials={real_user=slave}} blocked reactor thread for 46505.8us
I20240502 01:54:30.656255 25056 rebalancer_tool.cc:756] re-synchronizing cluster state
E20240502 01:54:31.057924 28638 ksck_remote.cc:634] unable to open table rebalancer_dynamic_table: Not found: the table does not exist: table_name: "rebalancer_dynamic_table"
I20240502 01:54:32.421290 25056 rebalancer_tool.cc:1136] tablet 0f8200b044ef44e1a0451baf2b1485d2: 'c2a28baa7ee2469294d94b7fe5455cff' -> 'd7449bb1df214bf4a5f93e4631312f08' move completed: OK
I20240502 01:54:32.439131 25056 rebalancer_tool.cc:1136] tablet ec67f45b2b604c129d8572c5ca6acbc4: 'c64840657cdf4b93980bd34120c94d9a' -> '0c64d40b5d7644bd8b29feaa80eab72f' move completed: OK
I20240502 01:54:32.450069 25056 rebalancer_tool.cc:1136] tablet db96ccbe81564bb589b0de65977b3321: 'c64840657cdf4b93980bd34120c94d9a' -> 'd7449bb1df214bf4a5f93e4631312f08' move completed: OK
I20240502 01:54:32.462026 25056 rebalancer_tool.cc:1136] tablet 166e717fb9624ae58adb6699790daf7a: 'c2a28baa7ee2469294d94b7fe5455cff' -> '28157bc3e678428fb5f93c077d528513' move completed: OK
I20240502 01:54:32.474088 25056 rebalancer_tool.cc:1136] tablet ad3cd6fff977431b9cd5df6d083bef80: '759257ea7bef415ca698e60d82199d77' -> '41c70f373c9d49a49891b03a12c6cb2b' move completed: OK
I20240502 01:54:32.489545 25056 rebalancer_tool.cc:1136] tablet d52f4b5c815f4188a09eb12ec5375361: 'c2a28baa7ee2469294d94b7fe5455cff' -> '28157bc3e678428fb5f93c077d528513' move completed: OK
I20240502 01:54:33.188220 25056 rebalancer_tool.cc:1136] tablet 4b924192d8da4a8bb7df2e6c5e4cbd4f: '759257ea7bef415ca698e60d82199d77' -> '41c70f373c9d49a49891b03a12c6cb2b' move completed: OK
I20240502 01:54:33.207801 25056 rebalancer_tool.cc:1136] tablet 992544bbea6547de96298fd32d74b232: 'd7449bb1df214bf4a5f93e4631312f08' -> '06543b05a16f42099bc86db2a0284412' move completed: OK
I20240502 01:54:33.219044 25056 rebalancer_tool.cc:1136] tablet b60b948dae4841dd9c0919f2fcac5299: '0c64d40b5d7644bd8b29feaa80eab72f' -> 'c64840657cdf4b93980bd34120c94d9a' move completed: OK
I20240502 01:54:34.104185 25056 rebalancer_tool.cc:1136] tablet e00feee2662a48d8a1a57f9e52275154: '0c64d40b5d7644bd8b29feaa80eab72f' -> 'c64840657cdf4b93980bd34120c94d9a' move completed: OK
I20240502 01:54:34.117316 25056 rebalancer_tool.cc:1136] tablet daaadb773202454894890c3379c4a58d: 'c2a28baa7ee2469294d94b7fe5455cff' -> '41c70f373c9d49a49891b03a12c6cb2b' move completed: OK
I20240502 01:54:34.982197 25056 rebalancer_tool.cc:1136] tablet 3cf315f0117444ae9c5390a1e19460d9: '06543b05a16f42099bc86db2a0284412' -> '0c64d40b5d7644bd8b29feaa80eab72f' move completed: OK
I20240502 01:54:34.992097 25056 rebalancer_tool.cc:1136] tablet fedbec8860904a2cb5294037b7a4bcad: '759257ea7bef415ca698e60d82199d77' -> '41c70f373c9d49a49891b03a12c6cb2b' move completed: OK
I20240502 01:54:35.003196 25056 rebalancer_tool.cc:1136] tablet 0eed9ee06a494588a32b4c951d6d3a14: '06543b05a16f42099bc86db2a0284412' -> '28157bc3e678428fb5f93c077d528513' move completed: OK
I20240502 01:54:35.016121 25056 rebalancer_tool.cc:1136] tablet 9bceab60366b4ade98f86072d391e7f4: '759257ea7bef415ca698e60d82199d77' -> '0c64d40b5d7644bd8b29feaa80eab72f' move completed: OK
I20240502 01:54:35.031493 25056 rebalancer_tool.cc:1136] tablet 766241df5ee24264a5cb7176e6c97b91: '06543b05a16f42099bc86db2a0284412' -> '28157bc3e678428fb5f93c077d528513' move completed: OK
I20240502 01:54:35.042759 25056 rebalancer_tool.cc:1136] tablet 03fb607ef0d9469daca1d6f3c5259fb2: '28157bc3e678428fb5f93c077d528513' -> '759257ea7bef415ca698e60d82199d77' move completed: OK
I20240502 01:54:35.054365 25056 rebalancer_tool.cc:1136] tablet d4b6dfe0e8da4352b136d6f97d473bad: 'c2a28baa7ee2469294d94b7fe5455cff' -> 'd7449bb1df214bf4a5f93e4631312f08' move completed: OK
W20240502 01:54:35.964778 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 543ab4cdef864690bedccba59308edc5: could not find any suitable replica to move from server d7449bb1df214bf4a5f93e4631312f08 to server c64840657cdf4b93980bd34120c94d9a
W20240502 01:54:35.965752 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 868965c86d0d413fbbeaca41748259c6: could not find any suitable replica to move from server c2a28baa7ee2469294d94b7fe5455cff to server 0c64d40b5d7644bd8b29feaa80eab72f
W20240502 01:54:35.965868 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 714c5d94b5b64cb29dc398713b1727fe: could not find any suitable replica to move from server c64840657cdf4b93980bd34120c94d9a to server d7449bb1df214bf4a5f93e4631312f08
W20240502 01:54:35.966243 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b462b431b51b4d45baf0ff5a495aa90a: could not find any suitable replica to move from server 759257ea7bef415ca698e60d82199d77 to server 0c64d40b5d7644bd8b29feaa80eab72f
W20240502 01:54:35.966532 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 543ab4cdef864690bedccba59308edc5: could not find any suitable replica to move from server d7449bb1df214bf4a5f93e4631312f08 to server c2a28baa7ee2469294d94b7fe5455cff
W20240502 01:54:35.966619 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 869549e86fb24c718467c2625a66e8c1: could not find any suitable replica to move from server 28157bc3e678428fb5f93c077d528513 to server c64840657cdf4b93980bd34120c94d9a
W20240502 01:54:35.966688 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table adc0aa06508a406b91ce8b5a5a44ea29: could not find any suitable replica to move from server 0c64d40b5d7644bd8b29feaa80eab72f to server 759257ea7bef415ca698e60d82199d77
W20240502 01:54:35.966753 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 382119de47e04360a975948cedb418bc: could not find any suitable replica to move from server d7449bb1df214bf4a5f93e4631312f08 to server c64840657cdf4b93980bd34120c94d9a
W20240502 01:54:35.967064 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 06ae89337ff143b5a107c994cbecc25a: could not find any suitable replica to move from server 41c70f373c9d49a49891b03a12c6cb2b to server c2a28baa7ee2469294d94b7fe5455cff
W20240502 01:54:35.967478 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 048399628e3c4ff99f3a5d3a67b8ff83: could not find any suitable replica to move from server 759257ea7bef415ca698e60d82199d77 to server 06543b05a16f42099bc86db2a0284412
W20240502 01:54:35.968070 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca142102a7e44673b2cac39cee9b5bcd: could not find any suitable replica to move from server 0c64d40b5d7644bd8b29feaa80eab72f to server c64840657cdf4b93980bd34120c94d9a
W20240502 01:54:35.968183 25056 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f45904cc4cd41f0962b57ad839bb4c6: could not find any suitable replica to move from server 28157bc3e678428fb5f93c077d528513 to server 759257ea7bef415ca698e60d82199d77
I20240502 01:54:36.091034 25056 rebalancer_tool.cc:1074] tablet 0d246b118c1c4388ac04b3775ddbee89: 'c64840657cdf4b93980bd34120c94d9a' -> '41c70f373c9d49a49891b03a12c6cb2b' move scheduled
I20240502 01:54:36.145164 25056 rebalancer_tool.cc:1074] tablet 0fb2e5d929334665ae41e33056c520bc: '0c64d40b5d7644bd8b29feaa80eab72f' -> '28157bc3e678428fb5f93c077d528513' move scheduled
I20240502 01:54:36.207624 25056 rebalancer_tool.cc:1074] tablet 76a262a323dc486f80b91c54436eb8b7: '759257ea7bef415ca698e60d82199d77' -> '06543b05a16f42099bc86db2a0284412' move scheduled
I20240502 01:54:36.282611 25056 rebalancer_tool.cc:1074] tablet 0eed9ee06a494588a32b4c951d6d3a14: 'c2a28baa7ee2469294d94b7fe5455cff' -> 'd7449bb1df214bf4a5f93e4631312f08' move scheduled
I20240502 01:54:36.408658 25056 rebalancer_tool.cc:1074] tablet dcc8171c991b48ddbfda8007e3a87a38: 'c64840657cdf4b93980bd34120c94d9a' -> '41c70f373c9d49a49891b03a12c6cb2b' move scheduled
I20240502 01:54:36.485371 25056 rebalancer_tool.cc:1074] tablet fa8220d5cb8140e6a66a3b377df56fe4: '28157bc3e678428fb5f93c077d528513' -> '0c64d40b5d7644bd8b29feaa80eab72f' move scheduled
I20240502 01:54:36.551024 25056 rebalancer_tool.cc:1074] tablet 0b73e5695b28424ba127d1f434d1e88c: '06543b05a16f42099bc86db2a0284412' -> '759257ea7bef415ca698e60d82199d77' move scheduled
I20240502 01:54:36.662552 25056 rebalancer_tool.cc:1074] tablet 9bceab60366b4ade98f86072d391e7f4: 'c2a28baa7ee2469294d94b7fe5455cff' -> 'd7449bb1df214bf4a5f93e4631312f08' move scheduled
I20240502 01:54:36.719449 25056 rebalancer_tool.cc:1074] tablet 3964f5f080f145a5923dac2f076637c8: 'c64840657cdf4b93980bd34120c94d9a' -> '41c70f373c9d49a49891b03a12c6cb2b' move scheduled
I20240502 01:54:36.899099 25056 rebalancer_tool.cc:1074] tablet 5ab9f7fe67aa41b5971d5df9e4b5d788: '28157bc3e678428fb5f93c077d528513' -> '06543b05a16f42099bc86db2a0284412' move scheduled
I20240502 01:54:37.140733 25056 rebalancer_tool.cc:1074] tablet 97c9eb341ce1470db791324a882a0271: '0c64d40b5d7644bd8b29feaa80eab72f' -> '759257ea7bef415ca698e60d82199d77' move scheduled
I20240502 01:54:37.280081 25056 rebalancer_tool.cc:1074] tablet ee09342ee0f54863a158cb76d2112086: 'd7449bb1df214bf4a5f93e4631312f08' -> 'c2a28baa7ee2469294d94b7fe5455cff' move scheduled
I20240502 01:54:37.553712 25056 rebalancer_tool.cc:1074] tablet d4b6dfe0e8da4352b136d6f97d473bad: 'c64840657cdf4b93980bd34120c94d9a' -> '41c70f373c9d49a49891b03a12c6cb2b' move scheduled
I20240502 01:54:37.713729 25056 rebalancer_tool.cc:1074] tablet 835d8839b7d84a82befac50dcc5a7c60: '28157bc3e678428fb5f93c077d528513' -> '759257ea7bef415ca698e60d82199d77' move scheduled
I20240502 01:54:37.968664 25056 rebalancer_tool.cc:1074] tablet 84d602fa045e4cfa9b698d3e92e5cf8f: '06543b05a16f42099bc86db2a0284412' -> '0c64d40b5d7644bd8b29feaa80eab72f' move scheduled
I20240502 01:54:38.050681 25056 rebalancer_tool.cc:1074] tablet d210f233014e4e80bd225256b27132a4: 'd7449bb1df214bf4a5f93e4631312f08' -> 'c2a28baa7ee2469294d94b7fe5455cff' move scheduled
I20240502 01:54:38.164698 25056 rebalancer_tool.cc:1074] tablet da66e78c2c5044b2924682be0b643b0a: 'c64840657cdf4b93980bd34120c94d9a' -> '41c70f373c9d49a49891b03a12c6cb2b' move scheduled
I20240502 01:54:38.271900 25056 rebalancer_tool.cc:1074] tablet 9b186753ceeb4a3c9f22391bb9a5b25f: '28157bc3e678428fb5f93c077d528513' -> '759257ea7bef415ca698e60d82199d77' move scheduled
I20240502 01:54:38.449779 25056 rebalancer_tool.cc:1074] tablet 6dad8caf4f9744eea7be75d0a4890f65: '0c64d40b5d7644bd8b29feaa80eab72f' -> '06543b05a16f42099bc86db2a0284412' move scheduled
I20240502 01:54:55.151623 25056 rebalancer_tool.cc:1136] tablet da66e78c2c5044b2924682be0b643b0a: 'c64840657cdf4b93980bd34120c94d9a' -> '41c70f373c9d49a49891b03a12c6cb2b' move completed: OK
I20240502 01:54:55.161962 25056 rebalancer_tool.cc:1136] tablet d210f233014e4e80bd225256b27132a4: 'd7449bb1df214bf4a5f93e4631312f08' -> 'c2a28baa7ee2469294d94b7fe5455cff' move completed: OK
I20240502 01:54:55.173740 25056 rebalancer_tool.cc:1136] tablet 84d602fa045e4cfa9b698d3e92e5cf8f: '06543b05a16f42099bc86db2a0284412' -> '0c64d40b5d7644bd8b29feaa80eab72f' move completed: OK
I20240502 01:54:55.184698 25056 rebalancer_tool.cc:1136] tablet 835d8839b7d84a82befac50dcc5a7c60: '28157bc3e678428fb5f93c077d528513' -> '759257ea7bef415ca698e60d82199d77' move completed: OK
I20240502 01:54:55.195397 25056 rebalancer_tool.cc:1136] tablet d4b6dfe0e8da4352b136d6f97d473bad: 'c64840657cdf4b93980bd34120c94d9a' -> '41c70f373c9d49a49891b03a12c6cb2b' move completed: OK
I20240502 01:54:56.046765 25056 rebalancer_tool.cc:1136] tablet ee09342ee0f54863a158cb76d2112086: 'd7449bb1df214bf4a5f93e4631312f08' -> 'c2a28baa7ee2469294d94b7fe5455cff' move completed: OK
I20240502 01:54:56.057104 25056 rebalancer_tool.cc:1136] tablet 97c9eb341ce1470db791324a882a0271: '0c64d40b5d7644bd8b29feaa80eab72f' -> '759257ea7bef415ca698e60d82199d77' move completed: OK
I20240502 01:54:56.069629 25056 rebalancer_tool.cc:1136] tablet 5ab9f7fe67aa41b5971d5df9e4b5d788: '28157bc3e678428fb5f93c077d528513' -> '06543b05a16f42099bc86db2a0284412' move completed: OK
I20240502 01:54:56.096853 25056 rebalancer_tool.cc:1136] tablet 9bceab60366b4ade98f86072d391e7f4: 'c2a28baa7ee2469294d94b7fe5455cff' -> 'd7449bb1df214bf4a5f93e4631312f08' move completed: OK
I20240502 01:54:56.108968 25056 rebalancer_tool.cc:1136] tablet 3964f5f080f145a5923dac2f076637c8: 'c64840657cdf4b93980bd34120c94d9a' -> '41c70f373c9d49a49891b03a12c6cb2b' move completed: OK
I20240502 01:54:56.737017 25056 rebalancer_tool.cc:1136] tablet dcc8171c991b48ddbfda8007e3a87a38: 'c64840657cdf4b93980bd34120c94d9a' -> '41c70f373c9d49a49891b03a12c6cb2b' move completed: OK
I20240502 01:54:56.748732 25056 rebalancer_tool.cc:1136] tablet 0eed9ee06a494588a32b4c951d6d3a14: 'c2a28baa7ee2469294d94b7fe5455cff' -> 'd7449bb1df214bf4a5f93e4631312f08' move completed: OK
I20240502 01:54:57.406481 25056 rebalancer_tool.cc:1136] tablet 76a262a323dc486f80b91c54436eb8b7: '759257ea7bef415ca698e60d82199d77' -> '06543b05a16f42099bc86db2a0284412' move completed: OK
I20240502 01:54:57.434073 25056 rebalancer_tool.cc:1136] tablet 0fb2e5d929334665ae41e33056c520bc: '0c64d40b5d7644bd8b29feaa80eab72f' -> '28157bc3e678428fb5f93c077d528513' move completed: OK
I20240502 01:54:57.467463 25056 rebalancer_tool.cc:1136] tablet 0d246b118c1c4388ac04b3775ddbee89: 'c64840657cdf4b93980bd34120c94d9a' -> '41c70f373c9d49a49891b03a12c6cb2b' move completed: OK
I20240502 01:54:57.543356 25056 rebalancer_tool.cc:1074] tablet 9ad2f9c27d284098a8e4f20ebc4b3d8c: 'c2a28baa7ee2469294d94b7fe5455cff' -> 'c64840657cdf4b93980bd34120c94d9a' move scheduled
I20240502 01:54:57.633345 25056 rebalancer_tool.cc:1074] tablet 2d6f1c51e76342429deb0ccbf4ad8ce2: 'd7449bb1df214bf4a5f93e4631312f08' -> '41c70f373c9d49a49891b03a12c6cb2b' move scheduled
I20240502 01:54:57.702345 25056 rebalancer_tool.cc:1074] tablet 5d40847a56704b33b771c3fd87d6dfae: 'c2a28baa7ee2469294d94b7fe5455cff' -> '41c70f373c9d49a49891b03a12c6cb2b' move scheduled
I20240502 01:54:57.781553 25056 rebalancer_tool.cc:1074] tablet d81f9cab671a4258aee1dd4ea4a5ee6b: 'c64840657cdf4b93980bd34120c94d9a' -> 'd7449bb1df214bf4a5f93e4631312f08' move scheduled
I20240502 01:54:57.868355 25056 rebalancer_tool.cc:1074] tablet 0f8200b044ef44e1a0451baf2b1485d2: '759257ea7bef415ca698e60d82199d77' -> '0c64d40b5d7644bd8b29feaa80eab72f' move scheduled
I20240502 01:54:57.976397 25056 rebalancer_tool.cc:1074] tablet a9fb0b24337a4175972db268aac5df34: '06543b05a16f42099bc86db2a0284412' -> 'c64840657cdf4b93980bd34120c94d9a' move scheduled
I20240502 01:54:58.055454 25056 rebalancer_tool.cc:1074] tablet 6e8d8fbd2d1249d5b5de5f189b0d03e9: '28157bc3e678428fb5f93c077d528513' -> 'c2a28baa7ee2469294d94b7fe5455cff' move scheduled
I20240502 01:54:58.123690 25056 rebalancer_tool.cc:1074] tablet fafa4ef17fac41b68e3bf4aaa8a4caa8: '41c70f373c9d49a49891b03a12c6cb2b' -> 'd7449bb1df214bf4a5f93e4631312f08' move scheduled
I20240502 01:54:58.179138 25056 rebalancer_tool.cc:1074] tablet 0a1b190978514cb49efb2b97cc97a65f: '0c64d40b5d7644bd8b29feaa80eab72f' -> '759257ea7bef415ca698e60d82199d77' move scheduled
I20240502 01:54:58.271271 25056 rebalancer_tool.cc:1074] tablet c351a8e007d04327bb14303d7b32d5be: 'c64840657cdf4b93980bd34120c94d9a' -> '06543b05a16f42099bc86db2a0284412' move scheduled
I20240502 01:54:58.519026 25056 rebalancer_tool.cc:1074] tablet 3237002dce97460799e219cf5b4968cb: '28157bc3e678428fb5f93c077d528513' -> 'c2a28baa7ee2469294d94b7fe5455cff' move scheduled
I20240502 01:54:58.737759 25056 rebalancer_tool.cc:1074] tablet ea103b58ddcc4459af3d51878a110d38: 'd7449bb1df214bf4a5f93e4631312f08' -> '41c70f373c9d49a49891b03a12c6cb2b' move scheduled
I20240502 01:54:58.893836 25056 rebalancer_tool.cc:1074] tablet 7ce01a3c9d9948ba8f153ad150cf3cb0: '0c64d40b5d7644bd8b29feaa80eab72f' -> 'c64840657cdf4b93980bd34120c94d9a' move scheduled
I20240502 01:54:58.999523 25056 rebalancer_tool.cc:1074] tablet 77a90f08d29a49d3b451f85cbdc828a6: '759257ea7bef415ca698e60d82199d77' -> '28157bc3e678428fb5f93c077d528513' move scheduled
I20240502 01:54:59.104427 25056 rebalancer_tool.cc:1074] tablet ba39439749a9486799373e173110f5c5: '06543b05a16f42099bc86db2a0284412' -> 'd7449bb1df214bf4a5f93e4631312f08' move scheduled
I20240502 01:54:59.168280 25056 rebalancer_tool.cc:1074] tablet e00feee2662a48d8a1a57f9e52275154: '41c70f373c9d49a49891b03a12c6cb2b' -> 'c2a28baa7ee2469294d94b7fe5455cff' move scheduled
I20240502 01:55:11.489079 25056 rebalancer_tool.cc:1136] tablet 6dad8caf4f9744eea7be75d0a4890f65: '0c64d40b5d7644bd8b29feaa80eab72f' -> '06543b05a16f42099bc86db2a0284412' move completed: OK
I20240502 01:55:11.534693 25056 rebalancer_tool.cc:1136] tablet 9b186753ceeb4a3c9f22391bb9a5b25f: '28157bc3e678428fb5f93c077d528513' -> '759257ea7bef415ca698e60d82199d77' move completed: OK
I20240502 01:55:12.267813 25056 rebalancer_tool.cc:1136] tablet 0b73e5695b28424ba127d1f434d1e88c: '06543b05a16f42099bc86db2a0284412' -> '759257ea7bef415ca698e60d82199d77' move completed: OK
I20240502 01:55:12.282919 25056 rebalancer_tool.cc:1136] tablet fa8220d5cb8140e6a66a3b377df56fe4: '28157bc3e678428fb5f93c077d528513' -> '0c64d40b5d7644bd8b29feaa80eab72f' move completed: OK
I20240502 01:55:12.387475 25056 rebalancer_tool.cc:1074] tablet 00c38c0987ed469eba1fb6f76ee2f6a9: '0c64d40b5d7644bd8b29feaa80eab72f' -> '06543b05a16f42099bc86db2a0284412' move scheduled
I20240502 01:55:12.447283 25056 rebalancer_tool.cc:1074] tablet 3a2f4ea104d4409383e5eb32ca2f78b0: '759257ea7bef415ca698e60d82199d77' -> '28157bc3e678428fb5f93c077d528513' move scheduled
I20240502 01:55:12.515194 25056 rebalancer_tool.cc:1074] tablet daaadb773202454894890c3379c4a58d: '759257ea7bef415ca698e60d82199d77' -> '28157bc3e678428fb5f93c077d528513' move scheduled
I20240502 01:55:15.265102 25056 rebalancer_tool.cc:1136] tablet 77a90f08d29a49d3b451f85cbdc828a6: '759257ea7bef415ca698e60d82199d77' -> '28157bc3e678428fb5f93c077d528513' move completed: OK
I20240502 01:55:15.285688 25056 rebalancer_tool.cc:1136] tablet ea103b58ddcc4459af3d51878a110d38: 'd7449bb1df214bf4a5f93e4631312f08' -> '41c70f373c9d49a49891b03a12c6cb2b' move completed: OK
I20240502 01:55:15.302886 25056 rebalancer_tool.cc:1136] tablet 3237002dce97460799e219cf5b4968cb: '28157bc3e678428fb5f93c077d528513' -> 'c2a28baa7ee2469294d94b7fe5455cff' move completed: OK
I20240502 01:55:15.328460 25056 rebalancer_tool.cc:1136] tablet 0a1b190978514cb49efb2b97cc97a65f: '0c64d40b5d7644bd8b29feaa80eab72f' -> '759257ea7bef415ca698e60d82199d77' move completed: OK
I20240502 01:55:15.356220 25056 rebalancer_tool.cc:1136] tablet fafa4ef17fac41b68e3bf4aaa8a4caa8: '41c70f373c9d49a49891b03a12c6cb2b' -> 'd7449bb1df214bf4a5f93e4631312f08' move completed: OK
I20240502 01:55:15.366966 25056 rebalancer_tool.cc:1136] tablet 7ce01a3c9d9948ba8f153ad150cf3cb0: '0c64d40b5d7644bd8b29feaa80eab72f' -> 'c64840657cdf4b93980bd34120c94d9a' move completed: OK
I20240502 01:55:15.377899 25056 rebalancer_tool.cc:1136] tablet 6e8d8fbd2d1249d5b5de5f189b0d03e9: '28157bc3e678428fb5f93c077d528513' -> 'c2a28baa7ee2469294d94b7fe5455cff' move completed: OK
I20240502 01:55:16.072099 25056 rebalancer_tool.cc:1136] tablet 0f8200b044ef44e1a0451baf2b1485d2: '759257ea7bef415ca698e60d82199d77' -> '0c64d40b5d7644bd8b29feaa80eab72f' move completed: OK
I20240502 01:55:16.084924 25056 rebalancer_tool.cc:1136] tablet c351a8e007d04327bb14303d7b32d5be: 'c64840657cdf4b93980bd34120c94d9a' -> '06543b05a16f42099bc86db2a0284412' move completed: OK
I20240502 01:55:16.097898 25056 rebalancer_tool.cc:1136] tablet d81f9cab671a4258aee1dd4ea4a5ee6b: 'c64840657cdf4b93980bd34120c94d9a' -> 'd7449bb1df214bf4a5f93e4631312f08' move completed: OK
I20240502 01:55:16.733872 25056 rebalancer_tool.cc:1136] tablet 2d6f1c51e76342429deb0ccbf4ad8ce2: 'd7449bb1df214bf4a5f93e4631312f08' -> '41c70f373c9d49a49891b03a12c6cb2b' move completed: OK
I20240502 01:55:16.743409 25056 rebalancer_tool.cc:1136] tablet e00feee2662a48d8a1a57f9e52275154: '41c70f373c9d49a49891b03a12c6cb2b' -> 'c2a28baa7ee2469294d94b7fe5455cff' move completed: OK
I20240502 01:55:16.753430 25056 rebalancer_tool.cc:1136] tablet 9ad2f9c27d284098a8e4f20ebc4b3d8c: 'c2a28baa7ee2469294d94b7fe5455cff' -> 'c64840657cdf4b93980bd34120c94d9a' move completed: OK
I20240502 01:55:16.809581 25056 rebalancer_tool.cc:1074] tablet b1c6c622638444bd854616902b8d91ad: '0c64d40b5d7644bd8b29feaa80eab72f' -> 'd7449bb1df214bf4a5f93e4631312f08' move scheduled
I20240502 01:55:16.973570 25056 rebalancer_tool.cc:1074] tablet 992544bbea6547de96298fd32d74b232: '41c70f373c9d49a49891b03a12c6cb2b' -> 'c2a28baa7ee2469294d94b7fe5455cff' move scheduled
I20240502 01:55:17.042500 25056 rebalancer_tool.cc:1074] tablet 8a0ae0d8e8784db9988b3469d318f7fb: 'c64840657cdf4b93980bd34120c94d9a' -> '28157bc3e678428fb5f93c077d528513' move scheduled
I20240502 01:55:17.107380 25056 rebalancer_tool.cc:1074] tablet c0877e287cb94abf98723449b9805f80: '759257ea7bef415ca698e60d82199d77' -> 'd7449bb1df214bf4a5f93e4631312f08' move scheduled
I20240502 01:55:17.151712 25056 rebalancer_tool.cc:1074] tablet 0cce126a8ba64fb2b40c4ddd03982f19: '0c64d40b5d7644bd8b29feaa80eab72f' -> '41c70f373c9d49a49891b03a12c6cb2b' move scheduled
I20240502 01:55:17.212646 25056 rebalancer_tool.cc:1074] tablet 5152c0bcae6946059900d52f0d33ac5b: 'c2a28baa7ee2469294d94b7fe5455cff' -> '759257ea7bef415ca698e60d82199d77' move scheduled
I20240502 01:55:17.265194 25056 rebalancer_tool.cc:1074] tablet 3be66f4b90824a21a5f41915c9041970: 'c64840657cdf4b93980bd34120c94d9a' -> '28157bc3e678428fb5f93c077d528513' move scheduled
I20240502 01:55:17.432173 25056 rebalancer_tool.cc:1074] tablet 3a27dc6af117475cb227e3526d2a9007: '06543b05a16f42099bc86db2a0284412' -> 'd7449bb1df214bf4a5f93e4631312f08' move scheduled
I20240502 01:55:17.507140 25056 rebalancer_tool.cc:1074] tablet 469abc1f70fc456b998461bd367dfc84: '41c70f373c9d49a49891b03a12c6cb2b' -> '0c64d40b5d7644bd8b29feaa80eab72f' move scheduled
I20240502 01:55:17.759734 25056 rebalancer_tool.cc:1074] tablet 3c81ce3a90f04b818186bb6e43259d13: '41c70f373c9d49a49891b03a12c6cb2b' -> 'c2a28baa7ee2469294d94b7fe5455cff' move scheduled
I20240502 01:55:17.840801 25056 rebalancer_tool.cc:1074] tablet a2c3475bcad94e3dad4be87376e500bc: 'c64840657cdf4b93980bd34120c94d9a' -> '759257ea7bef415ca698e60d82199d77' move scheduled
I20240502 01:55:18.068449 25056 rebalancer_tool.cc:1074] tablet 37827d24c61a465f88b679cd694df62f: '28157bc3e678428fb5f93c077d528513' -> 'd7449bb1df214bf4a5f93e4631312f08' move scheduled
I20240502 01:55:18.220319 25056 rebalancer_tool.cc:1074] tablet 8b377c7402194b42a8eabc747f668fb2: '0c64d40b5d7644bd8b29feaa80eab72f' -> 'c2a28baa7ee2469294d94b7fe5455cff' move scheduled
W20240502 01:55:20.904920 2650 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.14.212.197:36677, user_credentials={real_user=slave}} blocked reactor thread for 105008us
I20240502 01:55:29.243460 25056 rebalancer_tool.cc:1136] tablet 3a2f4ea104d4409383e5eb32ca2f78b0: '759257ea7bef415ca698e60d82199d77' -> '28157bc3e678428fb5f93c077d528513' move completed: OK
I20240502 01:55:29.254081 25056 rebalancer_tool.cc:1136] tablet 00c38c0987ed469eba1fb6f76ee2f6a9: '0c64d40b5d7644bd8b29feaa80eab72f' -> '06543b05a16f42099bc86db2a0284412' move completed: OK
I20240502 01:55:29.265053 25056 rebalancer_tool.cc:1136] tablet ba39439749a9486799373e173110f5c5: '06543b05a16f42099bc86db2a0284412' -> 'd7449bb1df214bf4a5f93e4631312f08' move completed: OK
I20240502 01:55:30.017326 25056 rebalancer_tool.cc:1136] tablet a9fb0b24337a4175972db268aac5df34: '06543b05a16f42099bc86db2a0284412' -> 'c64840657cdf4b93980bd34120c94d9a' move completed: OK
I20240502 01:55:30.030756 25056 rebalancer_tool.cc:1136] tablet 5d40847a56704b33b771c3fd87d6dfae: 'c2a28baa7ee2469294d94b7fe5455cff' -> '41c70f373c9d49a49891b03a12c6cb2b' move completed: OK
I20240502 01:55:30.093020 25056 rebalancer_tool.cc:1074] tablet 4a3f86de558243eea0664d120c62f182: '06543b05a16f42099bc86db2a0284412' -> 'd7449bb1df214bf4a5f93e4631312f08' move scheduled
I20240502 01:55:30.155395 25056 rebalancer_tool.cc:1074] tablet ba18c000cd3f42978ed8fd01a4ded60c: 'c64840657cdf4b93980bd34120c94d9a' -> '759257ea7bef415ca698e60d82199d77' move scheduled
I20240502 01:55:30.235241 25056 rebalancer_tool.cc:1074] tablet 458b19e5186f4ce79f618a10e101c605: '28157bc3e678428fb5f93c077d528513' -> '41c70f373c9d49a49891b03a12c6cb2b' move scheduled
I20240502 01:55:30.296702 25056 rebalancer_tool.cc:1074] tablet 88457f21ad4a4e00bb372479c13963ec: 'c2a28baa7ee2469294d94b7fe5455cff' -> '0c64d40b5d7644bd8b29feaa80eab72f' move scheduled
W20240502 01:55:30.875128 3830 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.14.212.195:38879, user_credentials={real_user=slave}} blocked reactor thread for 50778.7us
I20240502 01:55:34.314561 25056 rebalancer_tool.cc:1136] tablet a2c3475bcad94e3dad4be87376e500bc: 'c64840657cdf4b93980bd34120c94d9a' -> '759257ea7bef415ca698e60d82199d77' move completed: OK
I20240502 01:55:34.366024 25056 rebalancer_tool.cc:1136] tablet 3c81ce3a90f04b818186bb6e43259d13: '41c70f373c9d49a49891b03a12c6cb2b' -> 'c2a28baa7ee2469294d94b7fe5455cff' move completed: OK
I20240502 01:55:36.051359 25056 rebalancer_tool.cc:1136] tablet 3be66f4b90824a21a5f41915c9041970: 'c64840657cdf4b93980bd34120c94d9a' -> '28157bc3e678428fb5f93c077d528513' move completed: OK
I20240502 01:55:36.149804 25056 rebalancer_tool.cc:1136] tablet 5152c0bcae6946059900d52f0d33ac5b: 'c2a28baa7ee2469294d94b7fe5455cff' -> '759257ea7bef415ca698e60d82199d77' move completed: OK
I20240502 01:55:36.311503 25056 rebalancer_tool.cc:1136] tablet 0cce126a8ba64fb2b40c4ddd03982f19: '0c64d40b5d7644bd8b29feaa80eab72f' -> '41c70f373c9d49a49891b03a12c6cb2b' move completed: OK
I20240502 01:55:36.451617 25056 rebalancer_tool.cc:1136] tablet c0877e287cb94abf98723449b9805f80: '759257ea7bef415ca698e60d82199d77' -> 'd7449bb1df214bf4a5f93e4631312f08' move completed: OK
I20240502 01:55:36.541955 25056 rebalancer_tool.cc:1136] tablet 8a0ae0d8e8784db9988b3469d318f7fb: 'c64840657cdf4b93980bd34120c94d9a' -> '28157bc3e678428fb5f93c077d528513' move completed: OK
I20240502 01:55:37.705173 25056 rebalancer_tool.cc:1136] tablet b1c6c622638444bd854616902b8d91ad: '0c64d40b5d7644bd8b29feaa80eab72f' -> 'd7449bb1df214bf4a5f93e4631312f08' move completed: OK
I20240502 01:55:37.716611 25056 rebalancer_tool.cc:1136] tablet 3a27dc6af117475cb227e3526d2a9007: '06543b05a16f42099bc86db2a0284412' -> 'd7449bb1df214bf4a5f93e4631312f08' move completed: OK
I20240502 01:55:37.739984 25056 rebalancer_tool.cc:1136] tablet daaadb773202454894890c3379c4a58d: '759257ea7bef415ca698e60d82199d77' -> '28157bc3e678428fb5f93c077d528513' move completed: OK
I20240502 01:55:37.751041 25056 rebalancer_tool.cc:1136] tablet 8b377c7402194b42a8eabc747f668fb2: '0c64d40b5d7644bd8b29feaa80eab72f' -> 'c2a28baa7ee2469294d94b7fe5455cff' move completed: OK
I20240502 01:55:37.821239 25056 rebalancer_tool.cc:1074] tablet 5b3f40a9f1c34644b787905c28a54b29: 'c64840657cdf4b93980bd34120c94d9a' -> 'd7449bb1df214bf4a5f93e4631312f08' move scheduled
I20240502 01:55:37.912340 25056 rebalancer_tool.cc:1074] tablet feffa925599b4d6aafa48e09e4f6467a: 'd7449bb1df214bf4a5f93e4631312f08' -> '06543b05a16f42099bc86db2a0284412' move scheduled
I20240502 01:55:37.999943 25056 rebalancer_tool.cc:1074] tablet f018db81d6214bd0ac12feaf54541fed: '759257ea7bef415ca698e60d82199d77' -> '28157bc3e678428fb5f93c077d528513' move scheduled
I20240502 01:55:38.084868 25056 rebalancer_tool.cc:1074] tablet 36d358202b544e74bcc378060082df4d: 'c2a28baa7ee2469294d94b7fe5455cff' -> '0c64d40b5d7644bd8b29feaa80eab72f' move scheduled
I20240502 01:55:38.169111 25056 rebalancer_tool.cc:1074] tablet ad8bc5be633d4646a7c10d2483ed4e5b: '41c70f373c9d49a49891b03a12c6cb2b' -> 'c64840657cdf4b93980bd34120c94d9a' move scheduled
I20240502 01:55:38.233932 25056 rebalancer_tool.cc:1074] tablet 2b0e8db8c3414ce0a5ff12200fe6b544: 'd7449bb1df214bf4a5f93e4631312f08' -> '06543b05a16f42099bc86db2a0284412' move scheduled
I20240502 01:55:38.315589 25056 rebalancer_tool.cc:1074] tablet 2036e577d94242728ae4d00fe845121b: '759257ea7bef415ca698e60d82199d77' -> '28157bc3e678428fb5f93c077d528513' move scheduled
I20240502 01:55:38.473059 25056 rebalancer_tool.cc:1074] tablet 72f46ec6ed7848438cb9ad6e38053c4d: 'c64840657cdf4b93980bd34120c94d9a' -> '0c64d40b5d7644bd8b29feaa80eab72f' move scheduled
I20240502 01:55:39.573024 25056 rebalancer_tool.cc:1074] tablet 28dbb8d9ef5f4efdb5ba6e8ba9179aa8: '41c70f373c9d49a49891b03a12c6cb2b' -> '759257ea7bef415ca698e60d82199d77' move scheduled
I20240502 01:55:40.723556 25056 rebalancer_tool.cc:1074] tablet 41e68a88b92347e8855b6e5c82e7b7e1: '28157bc3e678428fb5f93c077d528513' -> 'c64840657cdf4b93980bd34120c94d9a' move scheduled
I20240502 01:55:44.661751 25056 rebalancer_tool.cc:1123] tablet 36d358202b544e74bcc378060082df4d: 'c2a28baa7ee2469294d94b7fe5455cff' -> '0c64d40b5d7644bd8b29feaa80eab72f' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:55:47.204001 25056 rebalancer_tool.cc:1136] tablet 458b19e5186f4ce79f618a10e101c605: '28157bc3e678428fb5f93c077d528513' -> '41c70f373c9d49a49891b03a12c6cb2b' move completed: OK
I20240502 01:55:47.217460 25056 rebalancer_tool.cc:1136] tablet ba18c000cd3f42978ed8fd01a4ded60c: 'c64840657cdf4b93980bd34120c94d9a' -> '759257ea7bef415ca698e60d82199d77' move completed: OK
I20240502 01:55:47.227777 25056 rebalancer_tool.cc:1136] tablet 4a3f86de558243eea0664d120c62f182: '06543b05a16f42099bc86db2a0284412' -> 'd7449bb1df214bf4a5f93e4631312f08' move completed: OK
I20240502 01:55:47.239630 25056 rebalancer_tool.cc:1136] tablet 37827d24c61a465f88b679cd694df62f: '28157bc3e678428fb5f93c077d528513' -> 'd7449bb1df214bf4a5f93e4631312f08' move completed: OK
I20240502 01:55:48.091147 25056 rebalancer_tool.cc:1136] tablet 469abc1f70fc456b998461bd367dfc84: '41c70f373c9d49a49891b03a12c6cb2b' -> '0c64d40b5d7644bd8b29feaa80eab72f' move completed: OK
I20240502 01:55:48.105115 25056 rebalancer_tool.cc:1136] tablet 992544bbea6547de96298fd32d74b232: '41c70f373c9d49a49891b03a12c6cb2b' -> 'c2a28baa7ee2469294d94b7fe5455cff' move completed: OK
I20240502 01:55:48.180262 25056 rebalancer_tool.cc:1074] tablet ae04b73f8a574264a26a647400a44235: '41c70f373c9d49a49891b03a12c6cb2b' -> '28157bc3e678428fb5f93c077d528513' move scheduled
I20240502 01:55:49.209221 25056 rebalancer_tool.cc:1074] tablet 2362b0bec07540f1bc0efedbcf61599c: 'd7449bb1df214bf4a5f93e4631312f08' -> '759257ea7bef415ca698e60d82199d77' move scheduled
I20240502 01:55:49.287467 25056 rebalancer_tool.cc:1074] tablet de3eb3ed981d4f629eaa7f473ea3b430: 'c64840657cdf4b93980bd34120c94d9a' -> '0c64d40b5d7644bd8b29feaa80eab72f' move scheduled
I20240502 01:55:50.353173 25056 rebalancer_tool.cc:1074] tablet bddb4a31d81d44afaabc247fd1449767: 'd7449bb1df214bf4a5f93e4631312f08' -> '28157bc3e678428fb5f93c077d528513' move scheduled
I20240502 01:55:54.491748 25056 rebalancer_tool.cc:1136] tablet 2036e577d94242728ae4d00fe845121b: '759257ea7bef415ca698e60d82199d77' -> '28157bc3e678428fb5f93c077d528513' move completed: OK
I20240502 01:55:55.257225 25056 rebalancer_tool.cc:1136] tablet ad8bc5be633d4646a7c10d2483ed4e5b: '41c70f373c9d49a49891b03a12c6cb2b' -> 'c64840657cdf4b93980bd34120c94d9a' move completed: OK
I20240502 01:55:55.269181 25056 rebalancer_tool.cc:1136] tablet f018db81d6214bd0ac12feaf54541fed: '759257ea7bef415ca698e60d82199d77' -> '28157bc3e678428fb5f93c077d528513' move completed: OK
I20240502 01:55:56.461179 25056 rebalancer_tool.cc:1136] tablet 88457f21ad4a4e00bb372479c13963ec: 'c2a28baa7ee2469294d94b7fe5455cff' -> '0c64d40b5d7644bd8b29feaa80eab72f' move completed: OK
I20240502 01:55:56.472914 25056 rebalancer_tool.cc:1136] tablet 72f46ec6ed7848438cb9ad6e38053c4d: 'c64840657cdf4b93980bd34120c94d9a' -> '0c64d40b5d7644bd8b29feaa80eab72f' move completed: OK
I20240502 01:55:56.539111 25056 rebalancer_tool.cc:1074] tablet 02a8e00ad2ad4bfa8bc1a21e50f821e4: '41c70f373c9d49a49891b03a12c6cb2b' -> 'c64840657cdf4b93980bd34120c94d9a' move scheduled
I20240502 01:56:56.646775 25056 rebalancer_tool.cc:1084] tablet db96ccbe81564bb589b0de65977b3321: '759257ea7bef415ca698e60d82199d77' -> '28157bc3e678428fb5f93c077d528513' move ignored: Timed out: ChangeConfig RPC to 127.14.212.194:44411 timed out after 60.000s (SENT)
I20240502 01:56:56.646955 25056 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): unacceptable health status UNAVAILABLE
I20240502 01:56:57.895489 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 21775
I20240502 01:56:57.905624 21897 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:56:57.909878 22054 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:57.910135 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:57.917017 22185 consensus_peers.cc:485] T 458b19e5186f4ce79f618a10e101c605 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:57.931702 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:57.986482 22054 consensus_peers.cc:485] T 0cce126a8ba64fb2b40c4ddd03982f19 P c64840657cdf4b93980bd34120c94d9a -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:58.000993 21923 consensus_peers.cc:485] T d4b6dfe0e8da4352b136d6f97d473bad P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:56:58.009810 21923 consensus_peers.cc:462] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:56:58.051949 24537 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:58.067135 24537 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:56:58.123824 7060 raft_consensus.cc:482] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:58.124011 7060 raft_consensus.cc:504] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:58.124675 7060 leader_election.cc:288] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:58.126749 22054 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:58.126946 22054 leader_election.cc:334] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:58.127053 22054 leader_election.cc:302] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c64840657cdf4b93980bd34120c94d9a; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:58.127297 7060 raft_consensus.cc:2726] T 77a90f08d29a49d3b451f85cbdc828a6 P c64840657cdf4b93980bd34120c94d9a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:58.150084 24537 consensus_peers.cc:485] T 42957eb0c3864381b67438035ec91c28 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:56:58.163383 21923 consensus_peers.cc:485] T 37827d24c61a465f88b679cd694df62f P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:56:58.168133 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:58.181957 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:58.230978 24537 consensus_peers.cc:485] T 6e4e454146f24104adf99041a0e5a3ad P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:58.291716 24537 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:58.332489 24934 consensus_peers.cc:485] T 1962a92af64a4249ab5a041458094936 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:56:58.367704 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 21775
I20240502 01:56:58.371783 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling FlushMRSOp(5d40847a56704b33b771c3fd87d6dfae): perf score=0.033347
W20240502 01:56:58.372992 22054 consensus_peers.cc:485] T 4a3f86de558243eea0664d120c62f182 P c64840657cdf4b93980bd34120c94d9a -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:56:58.401746 21923 consensus_peers.cc:485] T fafa4ef17fac41b68e3bf4aaa8a4caa8 P 06543b05a16f42099bc86db2a0284412 -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 01:56:58.438370 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: FlushMRSOp(5d40847a56704b33b771c3fd87d6dfae) complete. Timing: real 0.066s user 0.050s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":8423,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":10204,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":823,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":10421,"threads_started":2,"wal-append.queue_time_us":2048}
W20240502 01:56:58.438654 21922 connection.cc:535] client connection to 127.14.212.193:33785 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:58.439132 21924 connection.cc:535] server connection from 127.14.212.193:54117 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:58.439236 22184 connection.cc:535] client connection to 127.14.212.193:33785 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:56:58.439491 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 21907
W20240502 01:56:58.439798 22053 connection.cc:535] server connection from 127.14.212.193:56135 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:58.439992 22053 connection.cc:535] client connection to 127.14.212.193:33785 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:58.441008 24536 connection.cc:535] client connection to 127.14.212.193:33785 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:56:58.441293 25048 maintenance_manager.cc:419] P 41c70f373c9d49a49891b03a12c6cb2b: Scheduling UndoDeltaBlockGCOp(5d40847a56704b33b771c3fd87d6dfae): 4834 bytes on disk
W20240502 01:56:58.441817 24933 connection.cc:535] server connection from 127.14.212.193:44633 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:56:58.442493 24938 maintenance_manager.cc:643] P 41c70f373c9d49a49891b03a12c6cb2b: UndoDeltaBlockGCOp(5d40847a56704b33b771c3fd87d6dfae) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
W20240502 01:56:58.443126 22184 consensus_peers.cc:485] T da66e78c2c5044b2924682be0b643b0a P c2a28baa7ee2469294d94b7fe5455cff -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.443287 22184 consensus_peers.cc:485] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.443373 22184 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.443437 22184 consensus_peers.cc:485] T d81f9cab671a4258aee1dd4ea4a5ee6b P c2a28baa7ee2469294d94b7fe5455cff -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.443522 22184 consensus_peers.cc:485] T ba18c000cd3f42978ed8fd01a4ded60c P c2a28baa7ee2469294d94b7fe5455cff -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.444160 22053 consensus_peers.cc:485] T 3cf315f0117444ae9c5390a1e19460d9 P c64840657cdf4b93980bd34120c94d9a -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.444254 21922 consensus_peers.cc:485] T 8a0ae0d8e8784db9988b3469d318f7fb P 06543b05a16f42099bc86db2a0284412 -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.444319 22053 consensus_peers.cc:485] T 166e717fb9624ae58adb6699790daf7a P c64840657cdf4b93980bd34120c94d9a -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.444357 21922 consensus_peers.cc:485] T 992544bbea6547de96298fd32d74b232 P 06543b05a16f42099bc86db2a0284412 -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.444439 22053 consensus_peers.cc:485] T 0fb2e5d929334665ae41e33056c520bc P c64840657cdf4b93980bd34120c94d9a -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.444454 21922 consensus_peers.cc:485] T db96ccbe81564bb589b0de65977b3321 P 06543b05a16f42099bc86db2a0284412 -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.444545 21922 consensus_peers.cc:485] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 06543b05a16f42099bc86db2a0284412 -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.444550 22053 consensus_peers.cc:485] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P c64840657cdf4b93980bd34120c94d9a -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.444633 21922 consensus_peers.cc:485] T ec67f45b2b604c129d8572c5ca6acbc4 P 06543b05a16f42099bc86db2a0284412 -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.444728 21922 consensus_peers.cc:485] T 03fb607ef0d9469daca1d6f3c5259fb2 P 06543b05a16f42099bc86db2a0284412 -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.445529 24536 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.445703 24536 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.445850 24536 consensus_peers.cc:485] T 7c6a655aeff34e1a89d91aae14510355 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.445988 24536 consensus_peers.cc:485] T 0b73e5695b28424ba127d1f434d1e88c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:56:58.453467 22030 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:56:58.481292 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:58.517143 24536 consensus_peers.cc:485] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.532550 24934 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:56:58.542621 24934 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:56:58.550956 22185 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:58.568850 24933 consensus_peers.cc:485] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:56:58.617192 24649 maintenance_manager.cc:419] P 0c64d40b5d7644bd8b29feaa80eab72f: Scheduling FlushMRSOp(0f8200b044ef44e1a0451baf2b1485d2): perf score=0.033350
W20240502 01:56:58.618319 24537 consensus_peers.cc:485] T 2362b0bec07540f1bc0efedbcf61599c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:56:58.659289 22185 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:56:58.661731 24934 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 01:56:58.678741 24540 maintenance_manager.cc:643] P 0c64d40b5d7644bd8b29feaa80eab72f: FlushMRSOp(0f8200b044ef44e1a0451baf2b1485d2) complete. Timing: real 0.061s user 0.054s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":353,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1001,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":624,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":499,"threads_started":2,"wal-append.queue_time_us":342}
I20240502 01:56:58.681303 24649 maintenance_manager.cc:419] P 0c64d40b5d7644bd8b29feaa80eab72f: Scheduling UndoDeltaBlockGCOp(0f8200b044ef44e1a0451baf2b1485d2): 4834 bytes on disk
I20240502 01:56:58.681948 24540 maintenance_manager.cc:643] P 0c64d40b5d7644bd8b29feaa80eab72f: UndoDeltaBlockGCOp(0f8200b044ef44e1a0451baf2b1485d2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
W20240502 01:56:58.691196 24933 consensus_peers.cc:485] T 835d8839b7d84a82befac50dcc5a7c60 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.721917 24934 consensus_peers.cc:485] T 6e1f54cbb35d45cd89ffd1a12eb2ec8b P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 01:56:58.726863 7112 raft_consensus.cc:482] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 28157bc3e678428fb5f93c077d528513)
I20240502 01:56:58.727049 7112 raft_consensus.cc:504] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:58.727684 7112 leader_election.cc:288] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:58.729647 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:56:58.729830 24537 leader_election.cc:334] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:58.729920 24537 leader_election.cc:302] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 28157bc3e678428fb5f93c077d528513, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:58.730161 7112 raft_consensus.cc:2726] T 5a4a927b8f1740208f99ab1aca81b67a P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:58.767395 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:56:58.769975 24933 consensus_peers.cc:485] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.807053 24934 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:56:58.908833 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 21907
W20240502 01:56:58.986428 24934 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:56:58.990680 24536 connection.cc:535] client connection to 127.14.212.194:44411 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:58.990756 24937 connection.cc:535] server connection from 127.14.212.194:34591 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:58.991238 22184 connection.cc:535] client connection to 127.14.212.194:44411 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:58.990680 24538 connection.cc:535] server connection from 127.14.212.194:42615 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:58.991745 22184 connection.cc:535] server connection from 127.14.212.194:49055 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:58.991822 22056 connection.cc:535] server connection from 127.14.212.194:34765 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:56:58.992291 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 22038
W20240502 01:56:58.992959 24933 connection.cc:535] client connection to 127.14.212.194:44411 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:58.994261 22184 consensus_peers.cc:485] T 3964f5f080f145a5923dac2f076637c8 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.994403 22184 consensus_peers.cc:485] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.994537 22184 consensus_peers.cc:485] T dcc8171c991b48ddbfda8007e3a87a38 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.994624 22184 consensus_peers.cc:485] T de3eb3ed981d4f629eaa7f473ea3b430 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.994706 22184 consensus_peers.cc:485] T 03fbd6c1568a44f38826a1c7ed9f7e99 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.994828 24933 consensus_peers.cc:485] T feffa925599b4d6aafa48e09e4f6467a P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.994937 24933 consensus_peers.cc:485] T cc0f7b64459848019ae2b748dd1ad8dc P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.994990 24933 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.995076 24933 consensus_peers.cc:485] T 6dad8caf4f9744eea7be75d0a4890f65 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.996902 22053 consensus_peers.cc:485] T fedbec8860904a2cb5294037b7a4bcad P c64840657cdf4b93980bd34120c94d9a -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.997074 22053 consensus_peers.cc:485] T 88457f21ad4a4e00bb372479c13963ec P c64840657cdf4b93980bd34120c94d9a -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.997177 22053 consensus_peers.cc:485] T 5a82a89e92d243a78373077ea5a1f5de P c64840657cdf4b93980bd34120c94d9a -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.997296 22053 consensus_peers.cc:485] T 00c38c0987ed469eba1fb6f76ee2f6a9 P c64840657cdf4b93980bd34120c94d9a -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.998231 24536 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.998435 24536 consensus_peers.cc:485] T fa8220d5cb8140e6a66a3b377df56fe4 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.998567 24536 consensus_peers.cc:485] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:58.998726 24536 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:56:59.005336 22161 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:56:59.009479 24537 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:56:59.069670 24933 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Couldn't send request to peer 06543b05a16f42099bc86db2a0284412. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.072638 22185 consensus_peers.cc:485] T 9b186753ceeb4a3c9f22391bb9a5b25f P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:56:59.085791 22185 consensus_peers.cc:485] T 54bb2c6e63de4207a86894acad10cfe3 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:56:59.147393 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:56:59.226444 7026 raft_consensus.cc:482] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 759257ea7bef415ca698e60d82199d77)
I20240502 01:56:59.226678 7026 raft_consensus.cc:504] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:56:59.227325 7026 leader_election.cc:288] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
W20240502 01:56:59.227434 22185 consensus_peers.cc:485] T b0c930e4ad4544c18bad9c0d02a16184 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:56:59.229084 24536 leader_election.cc:334] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111)
W20240502 01:56:59.229292 24536 leader_election.cc:334] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111)
I20240502 01:56:59.229415 24536 leader_election.cc:302] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77
I20240502 01:56:59.229815 7112 raft_consensus.cc:2726] T 72f46ec6ed7848438cb9ad6e38053c4d P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:56:59.263425 7156 raft_consensus.cc:482] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 759257ea7bef415ca698e60d82199d77)
I20240502 01:56:59.263653 7156 raft_consensus.cc:504] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:56:59.264576 7156 leader_election.cc:288] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:56:59.265092 22248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d602fa045e4cfa9b698d3e92e5cf8f" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "c2a28baa7ee2469294d94b7fe5455cff" is_pre_election: true
W20240502 01:56:59.266208 24536 leader_election.cc:334] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111)
I20240502 01:56:59.266327 24536 leader_election.cc:302] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff
I20240502 01:56:59.266644 7156 raft_consensus.cc:2726] T 84d602fa045e4cfa9b698d3e92e5cf8f P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:56:59.281352 7156 raft_consensus.cc:482] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 759257ea7bef415ca698e60d82199d77)
I20240502 01:56:59.281580 7156 raft_consensus.cc:504] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:56:59.282474 7156 leader_election.cc:288] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
I20240502 01:56:59.283005 24998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea103b58ddcc4459af3d51878a110d38" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
W20240502 01:56:59.283885 24536 leader_election.cc:334] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111)
I20240502 01:56:59.284024 24536 leader_election.cc:302] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 41c70f373c9d49a49891b03a12c6cb2b, 759257ea7bef415ca698e60d82199d77
I20240502 01:56:59.284281 7156 raft_consensus.cc:2726] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:59.294137 24537 consensus_peers.cc:485] T 5152c0bcae6946059900d52f0d33ac5b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:56:59.332451 7156 raft_consensus.cc:482] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 759257ea7bef415ca698e60d82199d77)
I20240502 01:56:59.332675 7156 raft_consensus.cc:504] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:56:59.333519 7156 leader_election.cc:288] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
W20240502 01:56:59.336720 24536 leader_election.cc:334] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111)
W20240502 01:56:59.336934 24536 leader_election.cc:334] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111)
I20240502 01:56:59.337042 24536 leader_election.cc:302] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77
I20240502 01:56:59.337291 7156 raft_consensus.cc:2726] T 5ab9f7fe67aa41b5971d5df9e4b5d788 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:59.370983 22185 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:56:59.383293 24537 consensus_peers.cc:485] T c351a8e007d04327bb14303d7b32d5be P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 01:56:59.403396 7156 raft_consensus.cc:482] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 759257ea7bef415ca698e60d82199d77)
I20240502 01:56:59.403640 7156 raft_consensus.cc:504] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:56:59.404387 7156 leader_election.cc:288] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:56:59.405551 24536 leader_election.cc:334] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111)
W20240502 01:56:59.406121 24537 leader_election.cc:334] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:59.406239 24537 leader_election.cc:302] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 759257ea7bef415ca698e60d82199d77, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:59.406672 7156 raft_consensus.cc:2726] T 5a16aff54e6f4975a97776c214890f57 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:56:59.420050 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 22038
W20240502 01:56:59.435382 22185 consensus_peers.cc:485] T 05bb6b1075f442aaa3bbe005c98ce610 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:56:59.444857 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:59.462827 22185 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:56:59.481838 24538 connection.cc:535] client connection to 127.14.212.195:38879 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:59.482569 22186 connection.cc:535] client connection to 127.14.212.195:38879 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:56:59.482802 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 22169
W20240502 01:56:59.483355 24936 connection.cc:535] client connection to 127.14.212.195:38879 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:59.484972 24538 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.195:38879: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.485159 24538 consensus_peers.cc:485] T 7812450aabc54e1db36799d8e030d1ed P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.195:38879: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.485268 24538 consensus_peers.cc:485] T b4225c66fa064750bc2c0f440e71d673 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.195:38879: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.485296 22186 consensus_peers.cc:485] T 3a27dc6af117475cb227e3526d2a9007 P c2a28baa7ee2469294d94b7fe5455cff -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.195:38879: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.485370 24538 consensus_peers.cc:485] T 2036e577d94242728ae4d00fe845121b P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.195:38879: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.485422 22186 consensus_peers.cc:485] T ad8bc5be633d4646a7c10d2483ed4e5b P c2a28baa7ee2469294d94b7fe5455cff -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.195:38879: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.485522 22186 consensus_peers.cc:485] T 3a2f4ea104d4409383e5eb32ca2f78b0 P c2a28baa7ee2469294d94b7fe5455cff -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.195:38879: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.485626 22186 consensus_peers.cc:485] T 41e68a88b92347e8855b6e5c82e7b7e1 P c2a28baa7ee2469294d94b7fe5455cff -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.195:38879: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.485656 24936 consensus_peers.cc:485] T 9b297d603934468a907a1e1e08b9c276 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.195:38879: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.485709 22186 consensus_peers.cc:485] T ba39439749a9486799373e173110f5c5 P c2a28baa7ee2469294d94b7fe5455cff -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.195:38879: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.485771 24936 consensus_peers.cc:485] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.195:38879: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.486554 24936 consensus_peers.cc:485] T 5d40847a56704b33b771c3fd87d6dfae P 41c70f373c9d49a49891b03a12c6cb2b -> Peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Couldn't send request to peer c64840657cdf4b93980bd34120c94d9a. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.195:38879: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:56:59.493091 22292 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:56:59.524868 24537 consensus_peers.cc:485] T 2b0e8db8c3414ce0a5ff12200fe6b544 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:56:59.533329 24934 consensus_peers.cc:485] T c41a6df530ce411db79aa3c582ffc869 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:56:59.567919 24537 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:56:59.612138 24537 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:56:59.667830 7156 raft_consensus.cc:482] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06543b05a16f42099bc86db2a0284412)
I20240502 01:56:59.668030 7156 raft_consensus.cc:504] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:56:59.668808 7156 leader_election.cc:288] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
W20240502 01:56:59.669983 24536 leader_election.cc:334] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111)
W20240502 01:56:59.670673 24537 leader_election.cc:334] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:59.670796 24537 leader_election.cc:302] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 06543b05a16f42099bc86db2a0284412, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:59.671077 7156 raft_consensus.cc:2726] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:59.671079 24537 leader_election.cc:334] T bddb4a31d81d44afaabc247fd1449767 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:59.703670 7156 raft_consensus.cc:482] T 83480ad46a764f29a4fdfd782151ee43 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06543b05a16f42099bc86db2a0284412)
I20240502 01:56:59.703936 7156 raft_consensus.cc:504] T 83480ad46a764f29a4fdfd782151ee43 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:56:59.705199 24998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83480ad46a764f29a4fdfd782151ee43" candidate_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "41c70f373c9d49a49891b03a12c6cb2b" is_pre_election: true
I20240502 01:56:59.706171 7156 leader_election.cc:288] T 83480ad46a764f29a4fdfd782151ee43 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200:37097)
W20240502 01:56:59.706336 24536 leader_election.cc:334] T 83480ad46a764f29a4fdfd782151ee43 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111)
I20240502 01:56:59.706777 24536 leader_election.cc:302] T 83480ad46a764f29a4fdfd782151ee43 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 06543b05a16f42099bc86db2a0284412, 41c70f373c9d49a49891b03a12c6cb2b
I20240502 01:56:59.707064 7156 raft_consensus.cc:2726] T 83480ad46a764f29a4fdfd782151ee43 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:59.721902 24933 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240502 01:56:59.752712 24536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111) [suppressed 236 similar messages]
W20240502 01:56:59.811667 24537 consensus_peers.cc:485] T 59056268198e4b8f9f4d2afe1132f9c8 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Couldn't send request to peer d7449bb1df214bf4a5f93e4631312f08. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 01:56:59.831130 7163 raft_consensus.cc:482] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06543b05a16f42099bc86db2a0284412)
I20240502 01:56:59.831370 7163 raft_consensus.cc:504] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } }
I20240502 01:56:59.832084 7163 leader_election.cc:288] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
W20240502 01:56:59.833395 24536 leader_election.cc:334] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111)
I20240502 01:56:59.867163 7150 raft_consensus.cc:482] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 759257ea7bef415ca698e60d82199d77)
I20240502 01:56:59.867345 7150 raft_consensus.cc:504] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:56:59.868088 7150 leader_election.cc:288] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:56:59.868983 7122 raft_consensus.cc:482] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 759257ea7bef415ca698e60d82199d77)
I20240502 01:56:59.869144 7122 raft_consensus.cc:504] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:56:59.869733 7122 leader_election.cc:288] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:56:59.869851 7122 raft_consensus.cc:482] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 759257ea7bef415ca698e60d82199d77)
I20240502 01:56:59.869961 7122 raft_consensus.cc:504] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:56:59.870599 7122 leader_election.cc:288] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:56:59.870740 7122 raft_consensus.cc:482] T c7ebb7d204f04c60a32c450e703e4690 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 759257ea7bef415ca698e60d82199d77)
I20240502 01:56:59.870877 7122 raft_consensus.cc:504] T c7ebb7d204f04c60a32c450e703e4690 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:56:59.871507 7122 leader_election.cc:288] T c7ebb7d204f04c60a32c450e703e4690 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
W20240502 01:56:59.871740 24933 leader_election.cc:334] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111)
W20240502 01:56:59.872416 24933 leader_election.cc:334] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111)
W20240502 01:56:59.872557 24933 leader_election.cc:334] T c7ebb7d204f04c60a32c450e703e4690 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111)
W20240502 01:56:59.872660 24933 leader_election.cc:334] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111)
W20240502 01:56:59.873015 24934 leader_election.cc:334] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:59.873313 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea103b58ddcc4459af3d51878a110d38" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" is_pre_election: true
I20240502 01:56:59.873553 24601 raft_consensus.cc:2454] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 1.
I20240502 01:56:59.873924 24934 leader_election.cc:302] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 759257ea7bef415ca698e60d82199d77, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:59.874197 7122 raft_consensus.cc:2726] T b1c6c622638444bd854616902b8d91ad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:56:59.874697 24934 leader_election.cc:302] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 41c70f373c9d49a49891b03a12c6cb2b; no voters: 759257ea7bef415ca698e60d82199d77
I20240502 01:56:59.874970 7122 raft_consensus.cc:2781] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:56:59.875070 7122 raft_consensus.cc:482] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 759257ea7bef415ca698e60d82199d77)
W20240502 01:56:59.875080 24934 leader_election.cc:334] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:59.875187 7122 raft_consensus.cc:3037] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:59.875264 24934 leader_election.cc:302] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 759257ea7bef415ca698e60d82199d77, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:59.875521 7164 raft_consensus.cc:2726] T 641ed1f2bf7e40d7a8a6472cf9cd5519 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:56:59.877326 7122 raft_consensus.cc:504] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:56:59.878257 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea103b58ddcc4459af3d51878a110d38" candidate_uuid: "41c70f373c9d49a49891b03a12c6cb2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0c64d40b5d7644bd8b29feaa80eab72f"
I20240502 01:56:59.878489 24601 raft_consensus.cc:3037] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:56:59.879849 24933 leader_election.cc:334] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111)
I20240502 01:56:59.880054 7122 leader_election.cc:288] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 election: Requested vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:56:59.881234 24601 raft_consensus.cc:2454] T ea103b58ddcc4459af3d51878a110d38 P 0c64d40b5d7644bd8b29feaa80eab72f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c70f373c9d49a49891b03a12c6cb2b in term 2.
I20240502 01:56:59.881620 24934 leader_election.cc:302] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f, 41c70f373c9d49a49891b03a12c6cb2b; no voters: 759257ea7bef415ca698e60d82199d77
I20240502 01:56:59.881878 7122 raft_consensus.cc:2781] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:56:59.882028 7122 raft_consensus.cc:686] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 LEADER]: Becoming Leader. State: Replica: 41c70f373c9d49a49891b03a12c6cb2b, State: Running, Role: LEADER
I20240502 01:56:59.882300 7122 consensus_queue.cc:237] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:56:59.884274 7163 raft_consensus.cc:482] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06543b05a16f42099bc86db2a0284412)
I20240502 01:56:59.884461 7163 raft_consensus.cc:504] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:56:59.884841 7156 raft_consensus.cc:482] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06543b05a16f42099bc86db2a0284412)
I20240502 01:56:59.885123 7163 leader_election.cc:288] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:56:59.885039 7156 raft_consensus.cc:504] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:56:59.886768 7156 leader_election.cc:288] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
W20240502 01:56:59.886947 24536 leader_election.cc:334] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111)
W20240502 01:56:59.887737 24536 leader_election.cc:334] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111)
W20240502 01:56:59.887915 24536 leader_election.cc:334] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111)
I20240502 01:56:59.887997 24536 leader_election.cc:302] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77
I20240502 01:56:59.888278 7156 raft_consensus.cc:2726] T ec67f45b2b604c129d8572c5ca6acbc4 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:59.889014 24537 leader_election.cc:334] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:56:59.889142 24537 leader_election.cc:302] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 06543b05a16f42099bc86db2a0284412, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:56:59.889386 7156 raft_consensus.cc:2726] T 0f8200b044ef44e1a0451baf2b1485d2 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:56:59.889197 21715 catalog_manager.cc:5484] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b reported cstate change: term changed from 1 to 2, leader changed from 759257ea7bef415ca698e60d82199d77 (127.14.212.193) to 41c70f373c9d49a49891b03a12c6cb2b (127.14.212.200). New cstate: current_term: 2 leader_uuid: "41c70f373c9d49a49891b03a12c6cb2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } }
I20240502 01:56:59.893633 7122 raft_consensus.cc:482] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 06543b05a16f42099bc86db2a0284412)
I20240502 01:56:59.893798 7122 raft_consensus.cc:504] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
W20240502 01:56:59.895701 24933 leader_election.cc:334] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111)
W20240502 01:56:59.895922 24934 leader_election.cc:334] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:56:59.896021 24934 leader_election.cc:302] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 06543b05a16f42099bc86db2a0284412, 28157bc3e678428fb5f93c077d528513
I20240502 01:56:59.896286 7164 raft_consensus.cc:2726] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:56:59.896315 7122 leader_election.cc:288] T 96c5ae723e7a43a0b91a743b0c192a67 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
W20240502 01:56:59.907241 22185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111) [suppressed 158 similar messages]
I20240502 01:56:59.909147 7042 raft_consensus.cc:482] T c7ebb7d204f04c60a32c450e703e4690 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 759257ea7bef415ca698e60d82199d77)
I20240502 01:56:59.909349 7042 raft_consensus.cc:504] T c7ebb7d204f04c60a32c450e703e4690 P c2a28baa7ee2469294d94b7fe5455cff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:56:59.910205 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 22169
W20240502 01:56:59.928835 24537 consensus_peers.cc:485] T 0bada775411248a3a7fbbeca2a9eea0c P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:56:59.992746 24538 connection.cc:535] client connection to 127.14.212.196:43093 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:59.993628 24538 connection.cc:535] server connection from 127.14.212.196:47685 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:56:59.994076 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 24521
W20240502 01:56:59.994525 24936 connection.cc:535] client connection to 127.14.212.196:43093 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:59.995834 24538 consensus_peers.cc:485] T c0877e287cb94abf98723449b9805f80 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Couldn't send request to peer c2a28baa7ee2469294d94b7fe5455cff. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.196:43093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.995985 24538 consensus_peers.cc:485] T 469abc1f70fc456b998461bd367dfc84 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Couldn't send request to peer c2a28baa7ee2469294d94b7fe5455cff. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.196:43093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.996110 24538 consensus_peers.cc:485] T 3237002dce97460799e219cf5b4968cb P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Couldn't send request to peer c2a28baa7ee2469294d94b7fe5455cff. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.196:43093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.996209 24538 leader_election.cc:334] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Network error: Client connection negotiation failed: client connection to 127.14.212.196:43093: connect: Connection refused (error 111)
I20240502 01:56:59.996325 24538 leader_election.cc:302] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c64d40b5d7644bd8b29feaa80eab72f; no voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff
I20240502 01:56:59.996634 7156 raft_consensus.cc:2726] T 3c81ce3a90f04b818186bb6e43259d13 P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:56:59.996717 24538 consensus_peers.cc:485] T 752d84f8d70e4bed91d0ea547923f22f P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Couldn't send request to peer c2a28baa7ee2469294d94b7fe5455cff. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.196:43093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.997076 24936 consensus_peers.cc:485] T f0622f8af0e045a4a3578b030bebdf4f P 41c70f373c9d49a49891b03a12c6cb2b -> Peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Couldn't send request to peer c2a28baa7ee2469294d94b7fe5455cff. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.196:43093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.997220 24936 consensus_peers.cc:485] T ee09342ee0f54863a158cb76d2112086 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Couldn't send request to peer c2a28baa7ee2469294d94b7fe5455cff. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.196:43093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.997319 24936 consensus_peers.cc:485] T 6e8d8fbd2d1249d5b5de5f189b0d03e9 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Couldn't send request to peer c2a28baa7ee2469294d94b7fe5455cff. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.196:43093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.997424 24936 consensus_peers.cc:485] T d210f233014e4e80bd225256b27132a4 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Couldn't send request to peer c2a28baa7ee2469294d94b7fe5455cff. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.196:43093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:59.997558 24936 leader_election.cc:334] T c7ebb7d204f04c60a32c450e703e4690 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Network error: Client connection negotiation failed: client connection to 127.14.212.196:43093: connect: Connection refused (error 111)
I20240502 01:56:59.997721 24936 leader_election.cc:302] T c7ebb7d204f04c60a32c450e703e4690 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 759257ea7bef415ca698e60d82199d77, c2a28baa7ee2469294d94b7fe5455cff
I20240502 01:56:59.998198 7122 raft_consensus.cc:2726] T c7ebb7d204f04c60a32c450e703e4690 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:57:00.002066 24642 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:57:00.060104 24934 consensus_peers.cc:485] T f018db81d6214bd0ac12feaf54541fed P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:57:00.125114 7164 raft_consensus.cc:482] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c64840657cdf4b93980bd34120c94d9a)
I20240502 01:57:00.125357 7164 raft_consensus.cc:504] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:57:00.126128 7164 leader_election.cc:288] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
W20240502 01:57:00.127727 24936 leader_election.cc:334] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Network error: Client connection negotiation failed: client connection to 127.14.212.195:38879: connect: Connection refused (error 111)
W20240502 01:57:00.128372 24934 leader_election.cc:334] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:57:00.128494 24934 leader_election.cc:302] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 28157bc3e678428fb5f93c077d528513, c64840657cdf4b93980bd34120c94d9a
I20240502 01:57:00.128772 7122 raft_consensus.cc:2726] T 0cce126a8ba64fb2b40c4ddd03982f19 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:57:00.183305 7122 raft_consensus.cc:482] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c64840657cdf4b93980bd34120c94d9a)
I20240502 01:57:00.183558 7122 raft_consensus.cc:504] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:57:00.184162 7122 leader_election.cc:288] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
W20240502 01:57:00.185631 24936 leader_election.cc:334] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Network error: Client connection negotiation failed: client connection to 127.14.212.195:38879: connect: Connection refused (error 111)
W20240502 01:57:00.185827 24936 leader_election.cc:334] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Network error: Client connection negotiation failed: client connection to 127.14.212.196:43093: connect: Connection refused (error 111)
I20240502 01:57:00.185931 24936 leader_election.cc:302] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a
I20240502 01:57:00.186188 7122 raft_consensus.cc:2726] T 4b924192d8da4a8bb7df2e6c5e4cbd4f P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:57:00.289649 7122 raft_consensus.cc:482] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c64840657cdf4b93980bd34120c94d9a)
I20240502 01:57:00.289873 7122 raft_consensus.cc:504] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } }
I20240502 01:57:00.290732 7122 leader_election.cc:288] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
W20240502 01:57:00.292028 24933 leader_election.cc:334] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111)
W20240502 01:57:00.292232 24934 leader_election.cc:334] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:57:00.292352 24934 leader_election.cc:302] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 759257ea7bef415ca698e60d82199d77, d7449bb1df214bf4a5f93e4631312f08
W20240502 01:57:00.292522 24936 leader_election.cc:334] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Network error: Client connection negotiation failed: client connection to 127.14.212.195:38879: connect: Connection refused (error 111)
I20240502 01:57:00.292620 7122 raft_consensus.cc:2726] T 28dbb8d9ef5f4efdb5ba6e8ba9179aa8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:57:00.295334 24933 consensus_peers.cc:485] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Couldn't send request to peer 759257ea7bef415ca698e60d82199d77. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:57:00.323319 7122 raft_consensus.cc:482] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c64840657cdf4b93980bd34120c94d9a)
I20240502 01:57:00.323609 7122 raft_consensus.cc:504] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:57:00.324285 7122 leader_election.cc:288] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
W20240502 01:57:00.325762 24936 leader_election.cc:334] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Network error: Client connection negotiation failed: client connection to 127.14.212.195:38879: connect: Connection refused (error 111)
W20240502 01:57:00.325953 24936 leader_election.cc:334] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Network error: Client connection negotiation failed: client connection to 127.14.212.196:43093: connect: Connection refused (error 111)
I20240502 01:57:00.326058 24936 leader_election.cc:302] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: c2a28baa7ee2469294d94b7fe5455cff, c64840657cdf4b93980bd34120c94d9a
I20240502 01:57:00.326314 7122 raft_consensus.cc:2726] T ad3cd6fff977431b9cd5df6d083bef80 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:57:00.362036 24934 consensus_peers.cc:485] T 76a262a323dc486f80b91c54436eb8b7 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:57:00.383927 7122 raft_consensus.cc:482] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06543b05a16f42099bc86db2a0284412)
I20240502 01:57:00.384156 7122 raft_consensus.cc:504] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:57:00.384527 7164 raft_consensus.cc:482] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06543b05a16f42099bc86db2a0284412)
I20240502 01:57:00.384716 7164 raft_consensus.cc:504] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } }
I20240502 01:57:00.385164 7122 leader_election.cc:288] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:57:00.385515 7164 leader_election.cc:288] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:57:00.386246 7164 raft_consensus.cc:482] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06543b05a16f42099bc86db2a0284412)
I20240502 01:57:00.386409 7164 raft_consensus.cc:504] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "759257ea7bef415ca698e60d82199d77" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 33785 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:57:00.386987 7164 leader_election.cc:288] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785)
I20240502 01:57:00.387696 7163 raft_consensus.cc:482] T 88457f21ad4a4e00bb372479c13963ec P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c64840657cdf4b93980bd34120c94d9a)
I20240502 01:57:00.387904 7163 raft_consensus.cc:504] T 88457f21ad4a4e00bb372479c13963ec P 0c64d40b5d7644bd8b29feaa80eab72f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } }
I20240502 01:57:00.388737 7163 leader_election.cc:288] T 88457f21ad4a4e00bb372479c13963ec P 0c64d40b5d7644bd8b29feaa80eab72f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
W20240502 01:57:00.398607 24537 consensus_peers.cc:485] T a9fb0b24337a4175972db268aac5df34 P 0c64d40b5d7644bd8b29feaa80eab72f -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:57:00.387109 7122 raft_consensus.cc:482] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06543b05a16f42099bc86db2a0284412)
W20240502 01:57:00.395139 24933 leader_election.cc:334] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111)
W20240502 01:57:00.399757 24933 leader_election.cc:334] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111)
W20240502 01:57:00.401451 24933 leader_election.cc:334] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111)
I20240502 01:57:00.401407 7122 raft_consensus.cc:504] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
W20240502 01:57:00.401890 24933 leader_election.cc:334] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 759257ea7bef415ca698e60d82199d77 (127.14.212.193:33785): Network error: Client connection negotiation failed: client connection to 127.14.212.193:33785: connect: Connection refused (error 111)
I20240502 01:57:00.402019 24933 leader_election.cc:302] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 06543b05a16f42099bc86db2a0284412, 759257ea7bef415ca698e60d82199d77
I20240502 01:57:00.402278 7122 leader_election.cc:288] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:57:00.402498 7122 raft_consensus.cc:2726] T 2d6f1c51e76342429deb0ccbf4ad8ce2 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:57:00.404387 24933 leader_election.cc:334] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111)
I20240502 01:57:00.404412 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 24521
I20240502 01:57:00.406191 7164 raft_consensus.cc:482] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06543b05a16f42099bc86db2a0284412)
I20240502 01:57:00.406371 7164 raft_consensus.cc:504] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } }
I20240502 01:57:00.407526 7164 leader_election.cc:288] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:57:00.407712 7164 raft_consensus.cc:482] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06543b05a16f42099bc86db2a0284412)
I20240502 01:57:00.407876 7164 raft_consensus.cc:504] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } }
I20240502 01:57:00.408826 7164 leader_election.cc:288] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
W20240502 01:57:00.409981 24933 leader_election.cc:334] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111)
W20240502 01:57:00.411558 24933 leader_election.cc:334] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111)
I20240502 01:57:00.412906 7164 raft_consensus.cc:482] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c64840657cdf4b93980bd34120c94d9a)
W20240502 01:57:00.412973 24934 leader_election.cc:334] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:57:00.413156 7186 raft_consensus.cc:482] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06543b05a16f42099bc86db2a0284412)
I20240502 01:57:00.413076 7164 raft_consensus.cc:504] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } } peers { permanent_uuid: "28157bc3e678428fb5f93c077d528513" member_type: VOTER last_known_addr { host: "127.14.212.198" port: 42375 } }
I20240502 01:57:00.413280 7186 raft_consensus.cc:504] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c64d40b5d7644bd8b29feaa80eab72f" member_type: VOTER last_known_addr { host: "127.14.212.197" port: 36677 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:57:00.413682 7164 leader_election.cc:288] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879), 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375)
I20240502 01:57:00.413782 7186 leader_election.cc:288] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:57:00.413086 24934 leader_election.cc:302] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 06543b05a16f42099bc86db2a0284412, d7449bb1df214bf4a5f93e4631312f08
W20240502 01:57:00.406349 24936 leader_election.cc:334] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Network error: Client connection negotiation failed: client connection to 127.14.212.195:38879: connect: Connection refused (error 111)
I20240502 01:57:00.414814 24936 leader_election.cc:302] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 06543b05a16f42099bc86db2a0284412, c64840657cdf4b93980bd34120c94d9a
W20240502 01:57:00.415338 24934 leader_election.cc:334] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
I20240502 01:57:00.415391 7164 raft_consensus.cc:2726] T d4b6dfe0e8da4352b136d6f97d473bad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:57:00.415472 24934 leader_election.cc:302] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 06543b05a16f42099bc86db2a0284412, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:57:00.415555 7164 raft_consensus.cc:2726] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:57:00.415695 7164 raft_consensus.cc:2726] T 37827d24c61a465f88b679cd694df62f P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:57:00.415932 24934 leader_election.cc:334] T 02a8e00ad2ad4bfa8bc1a21e50f821e4 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
W20240502 01:57:00.416144 24933 leader_election.cc:334] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111)
I20240502 01:57:00.416397 7186 raft_consensus.cc:482] T ac80dc6d73db4352b9864392badc16c1 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06543b05a16f42099bc86db2a0284412)
W20240502 01:57:00.416603 24934 leader_election.cc:334] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111)
I20240502 01:57:00.416558 7186 raft_consensus.cc:504] T ac80dc6d73db4352b9864392badc16c1 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:57:00.417296 7186 leader_election.cc:288] T ac80dc6d73db4352b9864392badc16c1 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
W20240502 01:57:00.418150 24936 leader_election.cc:334] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Network error: Client connection negotiation failed: client connection to 127.14.212.196:43093: connect: Connection refused (error 111)
I20240502 01:57:00.418274 24936 leader_election.cc:302] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff
W20240502 01:57:00.418926 24933 leader_election.cc:334] T ac80dc6d73db4352b9864392badc16c1 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411): Network error: Client connection negotiation failed: client connection to 127.14.212.194:44411: connect: Connection refused (error 111)
I20240502 01:57:00.419158 7186 raft_consensus.cc:2726] T 0d246b118c1c4388ac04b3775ddbee89 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:57:00.419435 24934 leader_election.cc:334] T ac80dc6d73db4352b9864392badc16c1 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737): Network error: Client connection negotiation failed: client connection to 127.14.212.199:35737: connect: Connection refused (error 111)
W20240502 01:57:00.419490 24936 leader_election.cc:334] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093): Network error: Client connection negotiation failed: client connection to 127.14.212.196:43093: connect: Connection refused (error 111)
I20240502 01:57:00.419597 24936 leader_election.cc:302] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 06543b05a16f42099bc86db2a0284412, c2a28baa7ee2469294d94b7fe5455cff
I20240502 01:57:00.419574 24934 leader_election.cc:302] T ac80dc6d73db4352b9864392badc16c1 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 06543b05a16f42099bc86db2a0284412, d7449bb1df214bf4a5f93e4631312f08
I20240502 01:57:00.419848 7186 raft_consensus.cc:2726] T 69f7f6d27ad1486393b60321f8efefb3 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:57:00.419976 24936 leader_election.cc:334] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879): Network error: Client connection negotiation failed: client connection to 127.14.212.195:38879: connect: Connection refused (error 111)
I20240502 01:57:00.420053 7186 raft_consensus.cc:2726] T ac80dc6d73db4352b9864392badc16c1 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:57:00.420102 24936 leader_election.cc:302] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 28157bc3e678428fb5f93c077d528513, c64840657cdf4b93980bd34120c94d9a
I20240502 01:57:00.420357 7187 raft_consensus.cc:2726] T daaadb773202454894890c3379c4a58d P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:57:00.438552 24934 consensus_peers.cc:485] T 37b77f3edaf3431a96551a9525197579 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 28157bc3e678428fb5f93c077d528513 (127.14.212.198:42375): Couldn't send request to peer 28157bc3e678428fb5f93c077d528513. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.198:42375: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:57:00.470165 24934 connection.cc:535] client connection to 127.14.212.197:36677 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:57:00.470805 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 24917
W20240502 01:57:00.471071 24934 leader_election.cc:334] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Network error: Client connection negotiation failed: client connection to 127.14.212.197:36677: connect: Connection refused (error 111)
I20240502 01:57:00.471205 24934 leader_election.cc:302] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41c70f373c9d49a49891b03a12c6cb2b; no voters: 06543b05a16f42099bc86db2a0284412, 0c64d40b5d7644bd8b29feaa80eab72f
I20240502 01:57:00.471532 7187 raft_consensus.cc:2726] T 83480ad46a764f29a4fdfd782151ee43 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:57:00.471586 24934 consensus_peers.cc:485] T ea103b58ddcc4459af3d51878a110d38 P 41c70f373c9d49a49891b03a12c6cb2b -> Peer 0c64d40b5d7644bd8b29feaa80eab72f (127.14.212.197:36677): Couldn't send request to peer 0c64d40b5d7644bd8b29feaa80eab72f. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.197:36677: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:57:00.477339 25042 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:57:00.845274 7122 raft_consensus.cc:482] T fedbec8860904a2cb5294037b7a4bcad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c64840657cdf4b93980bd34120c94d9a)
I20240502 01:57:00.845569 7122 raft_consensus.cc:504] T fedbec8860904a2cb5294037b7a4bcad P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "c64840657cdf4b93980bd34120c94d9a" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 38879 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:57:00.846508 7122 leader_election.cc:288] T fedbec8860904a2cb5294037b7a4bcad P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411), c64840657cdf4b93980bd34120c94d9a (127.14.212.195:38879)
I20240502 01:57:00.849323 7186 raft_consensus.cc:482] T 05bb6b1075f442aaa3bbe005c98ce610 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c2a28baa7ee2469294d94b7fe5455cff)
I20240502 01:57:00.849517 7186 raft_consensus.cc:504] T 05bb6b1075f442aaa3bbe005c98ce610 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:57:00.850193 7186 leader_election.cc:288] T 05bb6b1075f442aaa3bbe005c98ce610 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737)
I20240502 01:57:00.850196 7191 raft_consensus.cc:482] T 3964f5f080f145a5923dac2f076637c8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c2a28baa7ee2469294d94b7fe5455cff)
I20240502 01:57:00.852322 7191 raft_consensus.cc:504] T 3964f5f080f145a5923dac2f076637c8 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:57:00.853404 7186 raft_consensus.cc:482] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2a28baa7ee2469294d94b7fe5455cff)
I20240502 01:57:00.853767 7186 raft_consensus.cc:504] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7449bb1df214bf4a5f93e4631312f08" member_type: VOTER last_known_addr { host: "127.14.212.199" port: 35737 } } peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:57:00.854126 7191 leader_election.cc:288] T 3964f5f080f145a5923dac2f076637c8 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:57:00.854264 7191 raft_consensus.cc:482] T dcc8171c991b48ddbfda8007e3a87a38 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c2a28baa7ee2469294d94b7fe5455cff)
I20240502 01:57:00.854434 7186 leader_election.cc:288] T 458b19e5186f4ce79f618a10e101c605 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7449bb1df214bf4a5f93e4631312f08 (127.14.212.199:35737), c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093)
I20240502 01:57:00.854408 7191 raft_consensus.cc:504] T dcc8171c991b48ddbfda8007e3a87a38 P 41c70f373c9d49a49891b03a12c6cb2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2a28baa7ee2469294d94b7fe5455cff" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 43093 } } peers { permanent_uuid: "06543b05a16f42099bc86db2a0284412" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 44411 } } peers { permanent_uuid: "41c70f373c9d49a49891b03a12c6cb2b" member_type: VOTER last_known_addr { host: "127.14.212.200" port: 37097 } }
I20240502 01:57:00.854943 7191 leader_election.cc:288] T dcc8171c991b48ddbfda8007e3a87a38 P 41c70f373c9d49a49891b03a12c6cb2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2a28baa7ee2469294d94b7fe5455cff (127.14.212.196:43093), 06543b05a16f42099bc86db2a0284412 (127.14.212.194:44411)
I20240502 01:57:00.855891 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 24917
I20240502 01:57:00.919601 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 21684
I20240502 01:57:00.925823 21746 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:57:01.257892 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 21684
2024-05-02T01:57:01Z chronyd exiting
I20240502 01:57:01.332043 15187 test_util.cc:175] -----------------------------------------------
I20240502 01:57:01.332149 15187 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714614748785264-15187-0
[ FAILED ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0, where GetParam() = 4-byte object <00-00 00-00> (216572 ms)
[----------] 1 test from DDLDuringRebalancingTest (216572 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/0
I20240502 01:57:01.334684 15187 test_util.cc:268] Using random seed: -347738006
I20240502 01:57:01.336745 15187 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:57:01.336838 15187 ts_itest-base.cc:113] --------------
I20240502 01:57:01.336915 15187 ts_itest-base.cc:114] 8 tablet servers
I20240502 01:57:01.336988 15187 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:57:01.337038 15187 ts_itest-base.cc:116] --------------
2024-05-02T01:57:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:57:01Z Disabled control of system clock
I20240502 01:57:01.373282 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.254:39055
--webserver_interface=127.14.212.254
--webserver_port=0
--builtin_ntp_servers=127.14.212.212:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.254:39055
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 01:57:01.736518 7203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:01.736922 7203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:01.737059 7203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:01.744874 7203 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:57:01.745081 7203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:57:01.745182 7203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:01.745270 7203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:57:01.745339 7203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:57:01.755172 7203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.14.212.254:39055
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.254:39055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.212.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:57:01.756632 7203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:01.758524 7203 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:01.770404 7208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:01.770993 7209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:01.771426 7212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:01.772189 7203 server_base.cc:1034] running on GCE node
I20240502 01:57:01.773120 7203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:01.774713 7203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:01.775921 7203 hybrid_clock.cc:648] HybridClock initialized: now 1714615021775874 us; error 49 us; skew 500 ppm
I20240502 01:57:01.776285 7203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:57:01.782436 7203 webserver.cc:460] Webserver started at http://127.14.212.254:40437/ using document root <none> and password file <none>
I20240502 01:57:01.783293 7203 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:01.783430 7203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:01.783797 7203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:01.786703 7203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "88a1eca613b44e04bccf07120b70d16c"
format_stamp: "Formatted at 2024-05-02 01:57:01 on dist-test-slave-mk1r"
I20240502 01:57:01.787567 7203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "88a1eca613b44e04bccf07120b70d16c"
format_stamp: "Formatted at 2024-05-02 01:57:01 on dist-test-slave-mk1r"
I20240502 01:57:01.793220 7203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20240502 01:57:01.796959 7218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:01.798400 7203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:57:01.798573 7203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
uuid: "88a1eca613b44e04bccf07120b70d16c"
format_stamp: "Formatted at 2024-05-02 01:57:01 on dist-test-slave-mk1r"
I20240502 01:57:01.798769 7203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:01.816501 7203 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:01.817474 7203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:01.817770 7203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:01.837740 7203 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.254:39055
I20240502 01:57:01.837749 7269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.254:39055 every 8 connection(s)
I20240502 01:57:01.839242 7203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:57:01.843380 7270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:01.847849 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 7203
I20240502 01:57:01.848145 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:57:01.855299 7270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c: Bootstrap starting.
I20240502 01:57:01.859460 7270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:01.860956 7270 log.cc:824] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c: Log is configured to *not* fsync() on all Append() calls
I20240502 01:57:01.864789 7270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c: No bootstrap required, opened a new log
I20240502 01:57:01.871798 7270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a1eca613b44e04bccf07120b70d16c" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 39055 } }
I20240502 01:57:01.872229 7270 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:01.872352 7270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a1eca613b44e04bccf07120b70d16c, State: Initialized, Role: FOLLOWER
I20240502 01:57:01.873032 7270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a1eca613b44e04bccf07120b70d16c" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 39055 } }
I20240502 01:57:01.873262 7270 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:57:01.873391 7270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:57:01.873562 7270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:01.875999 7270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a1eca613b44e04bccf07120b70d16c" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 39055 } }
I20240502 01:57:01.876552 7270 leader_election.cc:302] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88a1eca613b44e04bccf07120b70d16c; no voters:
I20240502 01:57:01.877298 7270 leader_election.cc:288] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:57:01.877444 7275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:01.879917 7275 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c [term 1 LEADER]: Becoming Leader. State: Replica: 88a1eca613b44e04bccf07120b70d16c, State: Running, Role: LEADER
I20240502 01:57:01.880506 7275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a1eca613b44e04bccf07120b70d16c" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 39055 } }
I20240502 01:57:01.880797 7270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:57:01.883790 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88a1eca613b44e04bccf07120b70d16c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a1eca613b44e04bccf07120b70d16c" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 39055 } } }
I20240502 01:57:01.884275 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c [sys.catalog]: This master's current role is: LEADER
I20240502 01:57:01.884702 7278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88a1eca613b44e04bccf07120b70d16c. Latest consensus state: current_term: 1 leader_uuid: "88a1eca613b44e04bccf07120b70d16c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a1eca613b44e04bccf07120b70d16c" member_type: VOTER last_known_addr { host: "127.14.212.254" port: 39055 } } }
I20240502 01:57:01.885126 7278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c [sys.catalog]: This master's current role is: LEADER
I20240502 01:57:01.885201 7285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:57:01.891098 7285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:57:01.900825 7285 catalog_manager.cc:1347] Generated new cluster ID: 6fe3e55aae9d453181a3c9422c9c319c
I20240502 01:57:01.900992 7285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:57:01.917713 7285 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:57:01.919275 7285 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:57:01.931866 7285 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 88a1eca613b44e04bccf07120b70d16c: Generated new TSK 0
I20240502 01:57:01.933024 7285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:57:01.945382 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.193:0
--local_ip_for_outbound_sockets=127.14.212.193
--webserver_interface=127.14.212.193
--webserver_port=0
--tserver_master_addrs=127.14.212.254:39055
--builtin_ntp_servers=127.14.212.212:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:57:02.321462 7296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:02.321799 7296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:02.321947 7296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:02.329877 7296 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:57:02.330034 7296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:02.330217 7296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.193
I20240502 01:57:02.339993 7296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.212.193
--webserver_port=0
--tserver_master_addrs=127.14.212.254:39055
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.193
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:57:02.341511 7296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:02.343715 7296 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:02.356714 7301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:02.357185 7302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:02.357795 7304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:02.358470 7296 server_base.cc:1034] running on GCE node
I20240502 01:57:02.359138 7296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:02.360143 7296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:02.361354 7296 hybrid_clock.cc:648] HybridClock initialized: now 1714615022361305 us; error 36 us; skew 500 ppm
I20240502 01:57:02.361691 7296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:57:02.364163 7296 webserver.cc:460] Webserver started at http://127.14.212.193:41665/ using document root <none> and password file <none>
I20240502 01:57:02.365026 7296 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:02.365159 7296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:02.365509 7296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:02.370900 7296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "87e2815274bf41c4ace764c5e37599b5"
format_stamp: "Formatted at 2024-05-02 01:57:02 on dist-test-slave-mk1r"
I20240502 01:57:02.371731 7296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "87e2815274bf41c4ace764c5e37599b5"
format_stamp: "Formatted at 2024-05-02 01:57:02 on dist-test-slave-mk1r"
I20240502 01:57:02.377411 7296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20240502 01:57:02.381103 7310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:02.382637 7296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:57:02.382817 7296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "87e2815274bf41c4ace764c5e37599b5"
format_stamp: "Formatted at 2024-05-02 01:57:02 on dist-test-slave-mk1r"
I20240502 01:57:02.383034 7296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:02.405380 7296 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:02.406296 7296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:02.406627 7296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:02.407645 7296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:02.409257 7296 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:02.409375 7296 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:02.409493 7296 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:02.409575 7296 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:02.445417 7296 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.193:40913
I20240502 01:57:02.445487 7428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.193:40913 every 8 connection(s)
I20240502 01:57:02.446858 7296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:57:02.455485 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 7296
I20240502 01:57:02.455715 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:57:02.459204 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.194:0
--local_ip_for_outbound_sockets=127.14.212.194
--webserver_interface=127.14.212.194
--webserver_port=0
--tserver_master_addrs=127.14.212.254:39055
--builtin_ntp_servers=127.14.212.212:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:57:02.460839 7429 heartbeater.cc:347] Connected to a master server at 127.14.212.254:39055
I20240502 01:57:02.461217 7429 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:02.462005 7429 heartbeater.cc:511] Master 127.14.212.254:39055 requested a full tablet report, sending...
I20240502 01:57:02.464354 7235 ts_manager.cc:194] Registered new tserver with Master: 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:02.466676 7235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.193:36385
W20240502 01:57:02.812923 7433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:02.813339 7433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:02.813499 7433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:02.821295 7433 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:57:02.821460 7433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:02.821622 7433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.194
I20240502 01:57:02.831075 7433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.212.194
--webserver_port=0
--tserver_master_addrs=127.14.212.254:39055
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.194
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:57:02.832475 7433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:02.834329 7433 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:02.849048 7438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:02.849387 7441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:02.850023 7439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:02.850824 7433 server_base.cc:1034] running on GCE node
I20240502 01:57:02.851469 7433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:02.852541 7433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:02.853775 7433 hybrid_clock.cc:648] HybridClock initialized: now 1714615022853713 us; error 57 us; skew 500 ppm
I20240502 01:57:02.854117 7433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:57:02.856590 7433 webserver.cc:460] Webserver started at http://127.14.212.194:42589/ using document root <none> and password file <none>
I20240502 01:57:02.857420 7433 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:02.857544 7433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:02.857895 7433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:02.860713 7433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d599c2145ef344fbbe26469d4331175d"
format_stamp: "Formatted at 2024-05-02 01:57:02 on dist-test-slave-mk1r"
I20240502 01:57:02.861631 7433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d599c2145ef344fbbe26469d4331175d"
format_stamp: "Formatted at 2024-05-02 01:57:02 on dist-test-slave-mk1r"
I20240502 01:57:02.868561 7433 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20240502 01:57:02.872301 7447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:02.873849 7433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:57:02.874009 7433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d599c2145ef344fbbe26469d4331175d"
format_stamp: "Formatted at 2024-05-02 01:57:02 on dist-test-slave-mk1r"
I20240502 01:57:02.874223 7433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:02.904575 7433 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:02.905521 7433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:02.905807 7433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:02.906816 7433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:02.908423 7433 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:02.908515 7433 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:02.908636 7433 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:02.908707 7433 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:02.941871 7433 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.194:39047
I20240502 01:57:02.941948 7560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.194:39047 every 8 connection(s)
I20240502 01:57:02.943259 7433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:57:02.945964 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 7433
I20240502 01:57:02.946209 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:57:02.949966 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.195:0
--local_ip_for_outbound_sockets=127.14.212.195
--webserver_interface=127.14.212.195
--webserver_port=0
--tserver_master_addrs=127.14.212.254:39055
--builtin_ntp_servers=127.14.212.212:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:57:02.957381 7561 heartbeater.cc:347] Connected to a master server at 127.14.212.254:39055
I20240502 01:57:02.957799 7561 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:02.958851 7561 heartbeater.cc:511] Master 127.14.212.254:39055 requested a full tablet report, sending...
I20240502 01:57:02.960498 7235 ts_manager.cc:194] Registered new tserver with Master: d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:02.961351 7235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.194:60679
W20240502 01:57:03.307401 7564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:03.307727 7564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:03.307852 7564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:03.315657 7564 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:57:03.315807 7564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:03.315977 7564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.195
I20240502 01:57:03.325618 7564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.212.195
--webserver_port=0
--tserver_master_addrs=127.14.212.254:39055
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.195
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:57:03.326997 7564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:03.328871 7564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:03.341790 7570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:03.342303 7569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:03.342922 7572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:03.343686 7564 server_base.cc:1034] running on GCE node
I20240502 01:57:03.344328 7564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:03.345373 7564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:03.346603 7564 hybrid_clock.cc:648] HybridClock initialized: now 1714615023346525 us; error 44 us; skew 500 ppm
I20240502 01:57:03.347079 7564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:57:03.349723 7564 webserver.cc:460] Webserver started at http://127.14.212.195:35775/ using document root <none> and password file <none>
I20240502 01:57:03.350526 7564 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:03.350643 7564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:03.350992 7564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:03.353852 7564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e8314214a0484251964e4b83a87b6786"
format_stamp: "Formatted at 2024-05-02 01:57:03 on dist-test-slave-mk1r"
I20240502 01:57:03.354643 7564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e8314214a0484251964e4b83a87b6786"
format_stamp: "Formatted at 2024-05-02 01:57:03 on dist-test-slave-mk1r"
I20240502 01:57:03.360000 7564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 01:57:03.363809 7578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:03.365294 7564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:57:03.365483 7564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e8314214a0484251964e4b83a87b6786"
format_stamp: "Formatted at 2024-05-02 01:57:03 on dist-test-slave-mk1r"
I20240502 01:57:03.365720 7564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:03.381163 7564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:03.382083 7564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:03.382373 7564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:03.383363 7564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:03.385006 7564 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:03.385098 7564 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:03.385221 7564 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:03.385293 7564 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:03.419356 7564 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.195:35355
I20240502 01:57:03.419435 7692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.195:35355 every 8 connection(s)
I20240502 01:57:03.420735 7564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:57:03.429369 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 7564
I20240502 01:57:03.429574 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:57:03.433169 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.196:0
--local_ip_for_outbound_sockets=127.14.212.196
--webserver_interface=127.14.212.196
--webserver_port=0
--tserver_master_addrs=127.14.212.254:39055
--builtin_ntp_servers=127.14.212.212:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:57:03.434087 7693 heartbeater.cc:347] Connected to a master server at 127.14.212.254:39055
I20240502 01:57:03.434437 7693 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:03.435346 7693 heartbeater.cc:511] Master 127.14.212.254:39055 requested a full tablet report, sending...
I20240502 01:57:03.440977 7235 ts_manager.cc:194] Registered new tserver with Master: e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:03.442155 7235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.195:52335
I20240502 01:57:03.470281 7429 heartbeater.cc:503] Master 127.14.212.254:39055 was elected leader, sending a full tablet report...
W20240502 01:57:03.783203 7696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:03.783535 7696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:03.783689 7696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:03.791570 7696 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:57:03.791734 7696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:03.791898 7696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.196
I20240502 01:57:03.801265 7696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.212.196
--webserver_port=0
--tserver_master_addrs=127.14.212.254:39055
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.196
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:57:03.802645 7696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:03.804472 7696 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:03.817582 7701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:03.817998 7702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:03.818439 7704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:03.819182 7696 server_base.cc:1034] running on GCE node
I20240502 01:57:03.819810 7696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:03.820845 7696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:03.822114 7696 hybrid_clock.cc:648] HybridClock initialized: now 1714615023822068 us; error 40 us; skew 500 ppm
I20240502 01:57:03.822458 7696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:57:03.824975 7696 webserver.cc:460] Webserver started at http://127.14.212.196:36837/ using document root <none> and password file <none>
I20240502 01:57:03.825788 7696 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:03.825896 7696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:03.826248 7696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:03.829254 7696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "fb68cc03fe5644e0b4846a80d385c412"
format_stamp: "Formatted at 2024-05-02 01:57:03 on dist-test-slave-mk1r"
I20240502 01:57:03.830044 7696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "fb68cc03fe5644e0b4846a80d385c412"
format_stamp: "Formatted at 2024-05-02 01:57:03 on dist-test-slave-mk1r"
I20240502 01:57:03.835633 7696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20240502 01:57:03.839468 7710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:03.841127 7696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:57:03.841301 7696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "fb68cc03fe5644e0b4846a80d385c412"
format_stamp: "Formatted at 2024-05-02 01:57:03 on dist-test-slave-mk1r"
I20240502 01:57:03.841532 7696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:03.859553 7696 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:03.860481 7696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:03.860774 7696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:03.861845 7696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:03.863536 7696 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:03.863641 7696 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:03.863760 7696 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:03.863837 7696 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:03.899379 7696 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.196:34255
I20240502 01:57:03.899457 7824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.196:34255 every 8 connection(s)
I20240502 01:57:03.900734 7696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:57:03.909785 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 7696
I20240502 01:57:03.909971 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:57:03.913594 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.197:0
--local_ip_for_outbound_sockets=127.14.212.197
--webserver_interface=127.14.212.197
--webserver_port=0
--tserver_master_addrs=127.14.212.254:39055
--builtin_ntp_servers=127.14.212.212:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:57:03.914490 7825 heartbeater.cc:347] Connected to a master server at 127.14.212.254:39055
I20240502 01:57:03.914817 7825 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:03.915603 7825 heartbeater.cc:511] Master 127.14.212.254:39055 requested a full tablet report, sending...
I20240502 01:57:03.917100 7235 ts_manager.cc:194] Registered new tserver with Master: fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:03.917950 7235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.196:38781
I20240502 01:57:03.964458 7561 heartbeater.cc:503] Master 127.14.212.254:39055 was elected leader, sending a full tablet report...
W20240502 01:57:04.265771 7828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:04.266091 7828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:04.266225 7828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:04.274194 7828 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:57:04.274341 7828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:04.274541 7828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.197
I20240502 01:57:04.284381 7828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.212.197
--webserver_port=0
--tserver_master_addrs=127.14.212.254:39055
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.197
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:57:04.285885 7828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:04.287830 7828 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:04.301180 7833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:04.301661 7834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:04.302194 7836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:04.303028 7828 server_base.cc:1034] running on GCE node
I20240502 01:57:04.303737 7828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:04.304898 7828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:04.306159 7828 hybrid_clock.cc:648] HybridClock initialized: now 1714615024306110 us; error 41 us; skew 500 ppm
I20240502 01:57:04.306545 7828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:57:04.308979 7828 webserver.cc:460] Webserver started at http://127.14.212.197:42377/ using document root <none> and password file <none>
I20240502 01:57:04.309760 7828 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:04.309880 7828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:04.310202 7828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:04.313331 7828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "7c2c6be28a0c47398034d89687b2d81f"
format_stamp: "Formatted at 2024-05-02 01:57:04 on dist-test-slave-mk1r"
I20240502 01:57:04.314106 7828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "7c2c6be28a0c47398034d89687b2d81f"
format_stamp: "Formatted at 2024-05-02 01:57:04 on dist-test-slave-mk1r"
I20240502 01:57:04.319684 7828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 01:57:04.323324 7842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:04.324821 7828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:57:04.325022 7828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7c2c6be28a0c47398034d89687b2d81f"
format_stamp: "Formatted at 2024-05-02 01:57:04 on dist-test-slave-mk1r"
I20240502 01:57:04.325289 7828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:04.345517 7828 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:04.346428 7828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:04.346724 7828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:04.347779 7828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:04.349457 7828 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:04.349552 7828 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:04.349673 7828 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:04.349740 7828 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:04.384081 7828 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.197:43213
I20240502 01:57:04.384171 7956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.197:43213 every 8 connection(s)
I20240502 01:57:04.385591 7828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:57:04.388357 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 7828
I20240502 01:57:04.388599 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:57:04.392468 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.198:0
--local_ip_for_outbound_sockets=127.14.212.198
--webserver_interface=127.14.212.198
--webserver_port=0
--tserver_master_addrs=127.14.212.254:39055
--builtin_ntp_servers=127.14.212.212:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:57:04.400175 7957 heartbeater.cc:347] Connected to a master server at 127.14.212.254:39055
I20240502 01:57:04.400552 7957 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:04.401602 7957 heartbeater.cc:511] Master 127.14.212.254:39055 requested a full tablet report, sending...
I20240502 01:57:04.403337 7235 ts_manager.cc:194] Registered new tserver with Master: 7c2c6be28a0c47398034d89687b2d81f (127.14.212.197:43213)
I20240502 01:57:04.404222 7235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.197:42741
I20240502 01:57:04.445685 7693 heartbeater.cc:503] Master 127.14.212.254:39055 was elected leader, sending a full tablet report...
W20240502 01:57:04.748260 7960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:04.748610 7960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:04.748761 7960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:04.756333 7960 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:57:04.756529 7960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:04.756655 7960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.198
I20240502 01:57:04.765992 7960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.14.212.198
--webserver_port=0
--tserver_master_addrs=127.14.212.254:39055
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.198
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:57:04.767486 7960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:04.769269 7960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:04.782295 7966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:04.782824 7968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:04.783334 7965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:04.784157 7960 server_base.cc:1034] running on GCE node
I20240502 01:57:04.784852 7960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:04.786028 7960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:04.787281 7960 hybrid_clock.cc:648] HybridClock initialized: now 1714615024787223 us; error 32 us; skew 500 ppm
I20240502 01:57:04.787665 7960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:57:04.790143 7960 webserver.cc:460] Webserver started at http://127.14.212.198:44493/ using document root <none> and password file <none>
I20240502 01:57:04.790952 7960 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:04.791095 7960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:04.791437 7960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:04.794288 7960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "2c800a3f253f4206b0d61d9804accf57"
format_stamp: "Formatted at 2024-05-02 01:57:04 on dist-test-slave-mk1r"
I20240502 01:57:04.795092 7960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "2c800a3f253f4206b0d61d9804accf57"
format_stamp: "Formatted at 2024-05-02 01:57:04 on dist-test-slave-mk1r"
I20240502 01:57:04.800608 7960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 01:57:04.804375 7974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:04.805784 7960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:57:04.805963 7960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "2c800a3f253f4206b0d61d9804accf57"
format_stamp: "Formatted at 2024-05-02 01:57:04 on dist-test-slave-mk1r"
I20240502 01:57:04.806187 7960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:04.837949 7960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:04.838899 7960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:04.839191 7960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:04.840265 7960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:04.841928 7960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:04.842038 7960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:04.842173 7960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:04.842245 7960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:04.876642 7960 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.198:33529
I20240502 01:57:04.876719 8090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.198:33529 every 8 connection(s)
I20240502 01:57:04.878286 7960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:57:04.878894 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 7960
I20240502 01:57:04.879112 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:57:04.883035 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.199:0
--local_ip_for_outbound_sockets=127.14.212.199
--webserver_interface=127.14.212.199
--webserver_port=0
--tserver_master_addrs=127.14.212.254:39055
--builtin_ntp_servers=127.14.212.212:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:57:04.892993 8091 heartbeater.cc:347] Connected to a master server at 127.14.212.254:39055
I20240502 01:57:04.893337 8091 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:04.894131 8091 heartbeater.cc:511] Master 127.14.212.254:39055 requested a full tablet report, sending...
I20240502 01:57:04.895641 7235 ts_manager.cc:194] Registered new tserver with Master: 2c800a3f253f4206b0d61d9804accf57 (127.14.212.198:33529)
I20240502 01:57:04.896433 7235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.198:34875
I20240502 01:57:04.920686 7825 heartbeater.cc:503] Master 127.14.212.254:39055 was elected leader, sending a full tablet report...
W20240502 01:57:05.232093 8094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:05.232435 8094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:05.232579 8094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:05.240406 8094 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:57:05.240566 8094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:05.240733 8094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.199
I20240502 01:57:05.250286 8094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.14.212.199
--webserver_port=0
--tserver_master_addrs=127.14.212.254:39055
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.199
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:57:05.251672 8094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:05.253552 8094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:05.266396 8099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:05.266947 8100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:05.267390 8102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:05.268158 8094 server_base.cc:1034] running on GCE node
I20240502 01:57:05.268806 8094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:05.269901 8094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:05.271150 8094 hybrid_clock.cc:648] HybridClock initialized: now 1714615025271030 us; error 112 us; skew 500 ppm
I20240502 01:57:05.271509 8094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:57:05.274001 8094 webserver.cc:460] Webserver started at http://127.14.212.199:38583/ using document root <none> and password file <none>
I20240502 01:57:05.274814 8094 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:05.274941 8094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:05.275271 8094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:05.278189 8094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "7b3556854afb4b85a56c63ed84e63b02"
format_stamp: "Formatted at 2024-05-02 01:57:05 on dist-test-slave-mk1r"
I20240502 01:57:05.279043 8094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "7b3556854afb4b85a56c63ed84e63b02"
format_stamp: "Formatted at 2024-05-02 01:57:05 on dist-test-slave-mk1r"
I20240502 01:57:05.285053 8094 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240502 01:57:05.288800 8109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:05.290321 8094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:57:05.290513 8094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "7b3556854afb4b85a56c63ed84e63b02"
format_stamp: "Formatted at 2024-05-02 01:57:05 on dist-test-slave-mk1r"
I20240502 01:57:05.290745 8094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:05.313213 8094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:05.314128 8094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:05.314409 8094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:05.315434 8094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:05.317114 8094 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:05.317234 8094 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:05.317358 8094 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:05.317437 8094 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:05.351889 8094 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.199:33829
I20240502 01:57:05.351987 8225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.199:33829 every 8 connection(s)
I20240502 01:57:05.353310 8094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:57:05.358456 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 8094
I20240502 01:57:05.358693 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 01:57:05.362428 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.200:0
--local_ip_for_outbound_sockets=127.14.212.200
--webserver_interface=127.14.212.200
--webserver_port=0
--tserver_master_addrs=127.14.212.254:39055
--builtin_ntp_servers=127.14.212.212:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:57:05.368355 8226 heartbeater.cc:347] Connected to a master server at 127.14.212.254:39055
I20240502 01:57:05.368777 8226 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:05.369944 8226 heartbeater.cc:511] Master 127.14.212.254:39055 requested a full tablet report, sending...
I20240502 01:57:05.371814 7235 ts_manager.cc:194] Registered new tserver with Master: 7b3556854afb4b85a56c63ed84e63b02 (127.14.212.199:33829)
I20240502 01:57:05.373129 7235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.199:52661
I20240502 01:57:05.407477 7957 heartbeater.cc:503] Master 127.14.212.254:39055 was elected leader, sending a full tablet report...
W20240502 01:57:05.719890 8229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:05.720232 8229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:05.720328 8229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:05.727998 8229 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:57:05.728152 8229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:05.728272 8229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.200
I20240502 01:57:05.737977 8229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.14.212.200
--webserver_port=0
--tserver_master_addrs=127.14.212.254:39055
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.200
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:57:05.739432 8229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:05.741331 8229 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:05.754158 8235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:05.754685 8237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:05.755214 8234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:05.756052 8229 server_base.cc:1034] running on GCE node
I20240502 01:57:05.756691 8229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:05.757761 8229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:05.758986 8229 hybrid_clock.cc:648] HybridClock initialized: now 1714615025758931 us; error 45 us; skew 500 ppm
I20240502 01:57:05.759336 8229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:57:05.761979 8229 webserver.cc:460] Webserver started at http://127.14.212.200:43075/ using document root <none> and password file <none>
I20240502 01:57:05.762764 8229 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:05.762887 8229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:05.763234 8229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:05.766125 8229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "7db0a568953d4aff999ebde5b5e820aa"
format_stamp: "Formatted at 2024-05-02 01:57:05 on dist-test-slave-mk1r"
I20240502 01:57:05.766902 8229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "7db0a568953d4aff999ebde5b5e820aa"
format_stamp: "Formatted at 2024-05-02 01:57:05 on dist-test-slave-mk1r"
I20240502 01:57:05.772560 8229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240502 01:57:05.776387 8243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:05.777804 8229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:57:05.777973 8229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "7db0a568953d4aff999ebde5b5e820aa"
format_stamp: "Formatted at 2024-05-02 01:57:05 on dist-test-slave-mk1r"
I20240502 01:57:05.778196 8229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:05.806648 8229 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:05.807631 8229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:05.807922 8229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:05.809033 8229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:05.810707 8229 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:05.810802 8229 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:05.810930 8229 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:05.811028 8229 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:05.844507 8229 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.200:32919
I20240502 01:57:05.844559 8357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.200:32919 every 8 connection(s)
I20240502 01:57:05.846261 8229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:57:05.849313 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 8229
I20240502 01:57:05.849524 15187 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 01:57:05.860334 8358 heartbeater.cc:347] Connected to a master server at 127.14.212.254:39055
I20240502 01:57:05.860680 8358 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:05.861613 8358 heartbeater.cc:511] Master 127.14.212.254:39055 requested a full tablet report, sending...
I20240502 01:57:05.863206 7235 ts_manager.cc:194] Registered new tserver with Master: 7db0a568953d4aff999ebde5b5e820aa (127.14.212.200:32919)
I20240502 01:57:05.864015 7235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.200:49997
I20240502 01:57:05.866065 15187 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:57:05.885597 15187 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:57:05.885768 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 7828
I20240502 01:57:05.894748 7952 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:57:05.901543 8091 heartbeater.cc:503] Master 127.14.212.254:39055 was elected leader, sending a full tablet report...
I20240502 01:57:05.981563 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 7828
I20240502 01:57:06.000219 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 7960
I20240502 01:57:06.008816 8086 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:57:06.093703 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 7960
I20240502 01:57:06.112658 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 8094
I20240502 01:57:06.120644 8221 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:57:06.204450 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 8094
I20240502 01:57:06.221864 15187 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskITWpJv/build/asan/bin/kudu pid 8229
I20240502 01:57:06.230732 8353 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:57:06.315223 15187 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskITWpJv/build/asan/bin/kudu with pid 8229
I20240502 01:57:10.084668 7234 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57734:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:57:10.142155 7628 tablet_service.cc:1467] Processing CreateTablet for tablet e4bb4b288a4d4fd689b750da93d2fa07 (DEFAULT_TABLE table=rebalance_test_table_0 [id=10aeef02ce1a4f96a59799c8070fba74]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:10.143476 7627 tablet_service.cc:1467] Processing CreateTablet for tablet 7dad5d30968a4698abc6688c2a71cddf (DEFAULT_TABLE table=rebalance_test_table_0 [id=10aeef02ce1a4f96a59799c8070fba74]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:10.143678 7628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4bb4b288a4d4fd689b750da93d2fa07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:10.144177 7627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dad5d30968a4698abc6688c2a71cddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:10.144593 7759 tablet_service.cc:1467] Processing CreateTablet for tablet 8c557be7764d456fa5d8a7883223951c (DEFAULT_TABLE table=rebalance_test_table_0 [id=10aeef02ce1a4f96a59799c8070fba74]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:10.146114 7759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c557be7764d456fa5d8a7883223951c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:10.146729 7495 tablet_service.cc:1467] Processing CreateTablet for tablet 8c557be7764d456fa5d8a7883223951c (DEFAULT_TABLE table=rebalance_test_table_0 [id=10aeef02ce1a4f96a59799c8070fba74]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:10.147831 7758 tablet_service.cc:1467] Processing CreateTablet for tablet 7dad5d30968a4698abc6688c2a71cddf (DEFAULT_TABLE table=rebalance_test_table_0 [id=10aeef02ce1a4f96a59799c8070fba74]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:10.148280 7495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c557be7764d456fa5d8a7883223951c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:10.148384 7758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dad5d30968a4698abc6688c2a71cddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:10.153787 7494 tablet_service.cc:1467] Processing CreateTablet for tablet e4bb4b288a4d4fd689b750da93d2fa07 (DEFAULT_TABLE table=rebalance_test_table_0 [id=10aeef02ce1a4f96a59799c8070fba74]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:10.154304 7494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4bb4b288a4d4fd689b750da93d2fa07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:10.174693 8387 tablet_bootstrap.cc:492] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:10.176649 8388 tablet_bootstrap.cc:492] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:10.178421 8389 tablet_bootstrap.cc:492] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:10.179469 8387 tablet_bootstrap.cc:654] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:10.181314 8388 tablet_bootstrap.cc:654] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:10.182003 8387 log.cc:824] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786: Log is configured to *not* fsync() on all Append() calls
I20240502 01:57:10.182499 8389 tablet_bootstrap.cc:654] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:10.184746 8388 log.cc:824] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412: Log is configured to *not* fsync() on all Append() calls
I20240502 01:57:10.185914 7360 tablet_service.cc:1467] Processing CreateTablet for tablet e4bb4b288a4d4fd689b750da93d2fa07 (DEFAULT_TABLE table=rebalance_test_table_0 [id=10aeef02ce1a4f96a59799c8070fba74]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:10.187220 7360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4bb4b288a4d4fd689b750da93d2fa07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:10.189186 7358 tablet_service.cc:1467] Processing CreateTablet for tablet 7dad5d30968a4698abc6688c2a71cddf (DEFAULT_TABLE table=rebalance_test_table_0 [id=10aeef02ce1a4f96a59799c8070fba74]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:10.189399 7361 tablet_service.cc:1467] Processing CreateTablet for tablet 8c557be7764d456fa5d8a7883223951c (DEFAULT_TABLE table=rebalance_test_table_0 [id=10aeef02ce1a4f96a59799c8070fba74]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:10.189857 7358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dad5d30968a4698abc6688c2a71cddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:10.190173 7361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c557be7764d456fa5d8a7883223951c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:10.193444 8388 tablet_bootstrap.cc:492] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:10.193938 8388 ts_tablet_manager.cc:1367] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.007s
I20240502 01:57:10.195030 8389 log.cc:824] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d: Log is configured to *not* fsync() on all Append() calls
I20240502 01:57:10.207561 8388 raft_consensus.cc:348] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:10.208755 8388 raft_consensus.cc:374] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:10.209021 8388 raft_consensus.cc:729] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:10.210253 8388 consensus_queue.cc:260] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:10.213340 8389 tablet_bootstrap.cc:492] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:10.213853 8389 ts_tablet_manager.cc:1367] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.003s
I20240502 01:57:10.222765 8392 tablet_bootstrap.cc:492] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:10.223392 8388 ts_tablet_manager.cc:1398] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.029s user 0.012s sys 0.010s
I20240502 01:57:10.224011 8388 tablet_bootstrap.cc:492] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:10.224143 8387 tablet_bootstrap.cc:492] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:10.224529 8387 ts_tablet_manager.cc:1367] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.034s
I20240502 01:57:10.227116 8392 tablet_bootstrap.cc:654] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:10.227473 8388 tablet_bootstrap.cc:654] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:10.228533 8392 log.cc:824] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5: Log is configured to *not* fsync() on all Append() calls
I20240502 01:57:10.229303 8389 raft_consensus.cc:348] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:10.229976 8389 raft_consensus.cc:374] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:10.230172 8389 raft_consensus.cc:729] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:10.231192 8389 consensus_queue.cc:260] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:10.236011 8388 tablet_bootstrap.cc:492] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:10.236254 8388 ts_tablet_manager.cc:1367] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20240502 01:57:10.236296 8387 raft_consensus.cc:348] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:10.236912 8387 raft_consensus.cc:374] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:10.237085 8387 raft_consensus.cc:729] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:10.237501 8388 raft_consensus.cc:348] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:10.237790 8388 raft_consensus.cc:374] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:10.237936 8388 raft_consensus.cc:729] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:10.238095 8387 consensus_queue.cc:260] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:10.238260 8388 consensus_queue.cc:260] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:10.239104 8388 ts_tablet_manager.cc:1398] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:57:10.245571 8392 tablet_bootstrap.cc:492] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:10.246052 8392 ts_tablet_manager.cc:1367] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.008s
I20240502 01:57:10.247346 8387 ts_tablet_manager.cc:1398] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.023s user 0.016s sys 0.003s
I20240502 01:57:10.247887 8387 tablet_bootstrap.cc:492] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:10.248114 8389 ts_tablet_manager.cc:1398] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.034s user 0.012s sys 0.014s
I20240502 01:57:10.248590 8389 tablet_bootstrap.cc:492] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:10.249892 8387 tablet_bootstrap.cc:654] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:10.250424 8389 tablet_bootstrap.cc:654] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:10.256032 8392 raft_consensus.cc:348] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:10.256536 8387 tablet_bootstrap.cc:492] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:10.256573 8392 raft_consensus.cc:374] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:10.256726 8387 ts_tablet_manager.cc:1367] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20240502 01:57:10.256703 8392 raft_consensus.cc:729] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:10.256754 8389 tablet_bootstrap.cc:492] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:10.256927 8389 ts_tablet_manager.cc:1367] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 01:57:10.257545 8392 consensus_queue.cc:260] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:10.257714 8387 raft_consensus.cc:348] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:10.257932 8387 raft_consensus.cc:374] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:10.258037 8387 raft_consensus.cc:729] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:10.257959 8389 raft_consensus.cc:348] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:10.258332 8389 raft_consensus.cc:374] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:10.258349 8387 consensus_queue.cc:260] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:10.258467 8389 raft_consensus.cc:729] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:10.258960 8387 ts_tablet_manager.cc:1398] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:10.259308 8389 consensus_queue.cc:260] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:10.260512 8389 ts_tablet_manager.cc:1398] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:57:10.261245 8392 ts_tablet_manager.cc:1398] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20240502 01:57:10.261783 8392 tablet_bootstrap.cc:492] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:10.264591 8392 tablet_bootstrap.cc:654] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:10.266660 8392 tablet_bootstrap.cc:492] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:10.266819 8392 ts_tablet_manager.cc:1367] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 01:57:10.267710 8392 raft_consensus.cc:348] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:10.267889 8392 raft_consensus.cc:374] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:10.267975 8392 raft_consensus.cc:729] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:10.268211 8392 consensus_queue.cc:260] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:10.268707 8392 ts_tablet_manager.cc:1398] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:57:10.269059 8392 tablet_bootstrap.cc:492] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:10.270463 8392 tablet_bootstrap.cc:654] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:10.272949 8392 tablet_bootstrap.cc:492] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:10.273149 8392 ts_tablet_manager.cc:1367] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240502 01:57:10.273793 8392 raft_consensus.cc:348] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:10.273986 8392 raft_consensus.cc:374] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:10.274072 8392 raft_consensus.cc:729] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:10.274299 8392 consensus_queue.cc:260] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:10.274838 8392 ts_tablet_manager.cc:1398] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:57:10.287297 8394 raft_consensus.cc:482] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:10.287596 8394 raft_consensus.cc:504] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:10.289621 8394 leader_election.cc:288] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:10.296840 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dad5d30968a4698abc6688c2a71cddf" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786" is_pre_election: true
I20240502 01:57:10.296844 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dad5d30968a4698abc6688c2a71cddf" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5" is_pre_election: true
I20240502 01:57:10.297428 7382 raft_consensus.cc:2454] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 0.
I20240502 01:57:10.297426 7648 raft_consensus.cc:2454] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 0.
I20240502 01:57:10.298090 7714 leader_election.cc:302] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:10.298524 8394 raft_consensus.cc:2781] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:10.298696 8394 raft_consensus.cc:482] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:10.298808 8394 raft_consensus.cc:3037] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:10.300976 8394 raft_consensus.cc:504] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:10.301646 8394 leader_election.cc:288] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:10.302191 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dad5d30968a4698abc6688c2a71cddf" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:10.302268 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dad5d30968a4698abc6688c2a71cddf" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:10.302397 7648 raft_consensus.cc:3037] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:10.302544 7382 raft_consensus.cc:3037] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:10.304778 7648 raft_consensus.cc:2454] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 1.
I20240502 01:57:10.304970 7382 raft_consensus.cc:2454] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 1.
I20240502 01:57:10.305375 7714 leader_election.cc:302] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:10.305738 8394 raft_consensus.cc:2781] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:10.306213 8394 raft_consensus.cc:686] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 1 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:10.306702 8394 consensus_queue.cc:237] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:10.311283 7234 catalog_manager.cc:5484] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 1 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:10.319509 8401 raft_consensus.cc:482] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:10.319736 8401 raft_consensus.cc:504] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:10.321444 8401 leader_election.cc:288] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:10.329036 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4bb4b288a4d4fd689b750da93d2fa07" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786" is_pre_election: true
I20240502 01:57:10.329382 7648 raft_consensus.cc:2454] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 0.
I20240502 01:57:10.329679 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4bb4b288a4d4fd689b750da93d2fa07" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d" is_pre_election: true
I20240502 01:57:10.329942 7314 leader_election.cc:302] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:10.330165 7516 raft_consensus.cc:2454] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 0.
I20240502 01:57:10.330377 8401 raft_consensus.cc:2781] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:10.330510 8401 raft_consensus.cc:482] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:10.330626 8401 raft_consensus.cc:3037] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:10.332711 8401 raft_consensus.cc:504] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:10.333364 8401 leader_election.cc:288] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:10.333856 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4bb4b288a4d4fd689b750da93d2fa07" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:10.333838 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4bb4b288a4d4fd689b750da93d2fa07" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:10.334091 7648 raft_consensus.cc:3037] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:10.334066 7516 raft_consensus.cc:3037] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:10.336194 7648 raft_consensus.cc:2454] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 1.
I20240502 01:57:10.336373 7516 raft_consensus.cc:2454] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 1.
I20240502 01:57:10.336663 7314 leader_election.cc:302] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:10.337059 8401 raft_consensus.cc:2781] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:10.337502 8401 raft_consensus.cc:686] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:10.338053 8401 consensus_queue.cc:237] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:10.342404 7234 catalog_manager.cc:5484] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
I20240502 01:57:10.346408 8401 raft_consensus.cc:482] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:10.346630 8401 raft_consensus.cc:504] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:10.347380 8401 leader_election.cc:288] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:10.347980 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c557be7764d456fa5d8a7883223951c" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d" is_pre_election: true
I20240502 01:57:10.348276 7516 raft_consensus.cc:2454] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 0.
I20240502 01:57:10.348727 7313 leader_election.cc:302] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters:
I20240502 01:57:10.349072 8401 raft_consensus.cc:2781] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:10.349176 8401 raft_consensus.cc:482] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:10.349288 8401 raft_consensus.cc:3037] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:10.351297 8401 raft_consensus.cc:504] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:10.351995 8401 leader_election.cc:288] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:10.352458 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c557be7764d456fa5d8a7883223951c" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:10.352669 7516 raft_consensus.cc:3037] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:10.353838 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c557be7764d456fa5d8a7883223951c" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412" is_pre_election: true
I20240502 01:57:10.354276 7780 raft_consensus.cc:2454] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 0.
I20240502 01:57:10.354784 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c557be7764d456fa5d8a7883223951c" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:10.355063 7779 raft_consensus.cc:3037] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:10.355270 7516 raft_consensus.cc:2454] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 1.
I20240502 01:57:10.355726 7313 leader_election.cc:302] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters:
I20240502 01:57:10.356060 8401 raft_consensus.cc:2781] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:10.356227 8401 raft_consensus.cc:686] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:10.356527 8401 consensus_queue.cc:237] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:10.358059 7779 raft_consensus.cc:2454] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 1.
I20240502 01:57:10.359529 7234 catalog_manager.cc:5484] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
W20240502 01:57:10.407663 7826 tablet.cc:2369] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:57:10.428249 7694 tablet.cc:2369] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:57:10.451071 7562 tablet.cc:2369] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:57:10.454994 7430 tablet.cc:2369] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:57:10.741992 8401 consensus_queue.cc:1035] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:10.751969 8401 consensus_queue.cc:1035] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:57:10.769623 8413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:10.770010 8413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:10.778133 8413 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:57:10.799144 8401 consensus_queue.cc:1035] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:10.804338 8431 consensus_queue.cc:1035] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240502 01:57:10.829387 8394 consensus_queue.cc:1035] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:10.836568 8445 consensus_queue.cc:1035] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 267.347 ms
time per row: 0.133673 ms
I20240502 01:57:11.207536 7232 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57734:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:57:11.217588 7494 tablet_service.cc:1467] Processing CreateTablet for tablet 26522a1c09bf44aca91d0410a7459b26 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6d75df9d09c64b669ed9d3280536e870]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:11.217756 7627 tablet_service.cc:1467] Processing CreateTablet for tablet 26522a1c09bf44aca91d0410a7459b26 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6d75df9d09c64b669ed9d3280536e870]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:11.218144 7494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26522a1c09bf44aca91d0410a7459b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:11.218119 7628 tablet_service.cc:1467] Processing CreateTablet for tablet 71ee1cc0eacc45149367f418f4e42954 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6d75df9d09c64b669ed9d3280536e870]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:11.218266 7627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26522a1c09bf44aca91d0410a7459b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:11.218524 7628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71ee1cc0eacc45149367f418f4e42954. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:11.218451 7758 tablet_service.cc:1467] Processing CreateTablet for tablet 26522a1c09bf44aca91d0410a7459b26 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6d75df9d09c64b669ed9d3280536e870]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:11.218876 7758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26522a1c09bf44aca91d0410a7459b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:11.225546 7495 tablet_service.cc:1467] Processing CreateTablet for tablet 71ee1cc0eacc45149367f418f4e42954 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6d75df9d09c64b669ed9d3280536e870]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:11.225549 7358 tablet_service.cc:1467] Processing CreateTablet for tablet 29dce0c6f40a4358ab8855cf0c709862 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6d75df9d09c64b669ed9d3280536e870]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:11.226048 7358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29dce0c6f40a4358ab8855cf0c709862. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:11.226059 7495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71ee1cc0eacc45149367f418f4e42954. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:11.226171 7626 tablet_service.cc:1467] Processing CreateTablet for tablet 29dce0c6f40a4358ab8855cf0c709862 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6d75df9d09c64b669ed9d3280536e870]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:11.226640 7626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29dce0c6f40a4358ab8855cf0c709862. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:11.227099 7759 tablet_service.cc:1467] Processing CreateTablet for tablet 71ee1cc0eacc45149367f418f4e42954 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6d75df9d09c64b669ed9d3280536e870]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:11.227541 7759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71ee1cc0eacc45149367f418f4e42954. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:11.229558 7493 tablet_service.cc:1467] Processing CreateTablet for tablet 29dce0c6f40a4358ab8855cf0c709862 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6d75df9d09c64b669ed9d3280536e870]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:11.230233 7493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29dce0c6f40a4358ab8855cf0c709862. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:11.241470 8478 tablet_bootstrap.cc:492] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:11.242372 8479 tablet_bootstrap.cc:492] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:11.243675 8477 tablet_bootstrap.cc:492] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:11.246304 8478 tablet_bootstrap.cc:654] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:11.246965 8479 tablet_bootstrap.cc:654] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:11.248812 8477 tablet_bootstrap.cc:654] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:11.253425 8477 tablet_bootstrap.cc:492] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:11.253719 8477 ts_tablet_manager.cc:1367] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240502 01:57:11.253878 8480 tablet_bootstrap.cc:492] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:11.254997 8477 raft_consensus.cc:348] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:11.255445 8477 raft_consensus.cc:374] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:11.255630 8477 raft_consensus.cc:729] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:11.256083 8477 consensus_queue.cc:260] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:11.257041 8477 ts_tablet_manager.cc:1398] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:57:11.257349 8480 tablet_bootstrap.cc:654] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:11.257453 8477 tablet_bootstrap.cc:492] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:11.257592 8478 tablet_bootstrap.cc:492] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:11.257802 8478 ts_tablet_manager.cc:1367] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20240502 01:57:11.259232 8477 tablet_bootstrap.cc:654] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:11.260177 8478 raft_consensus.cc:348] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:11.260533 8478 raft_consensus.cc:374] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:11.260712 8478 raft_consensus.cc:729] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:11.261164 8478 consensus_queue.cc:260] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:11.262159 8478 ts_tablet_manager.cc:1398] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:57:11.262626 8478 tablet_bootstrap.cc:492] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:11.265013 8478 tablet_bootstrap.cc:654] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:11.265321 8479 tablet_bootstrap.cc:492] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:11.266870 8479 ts_tablet_manager.cc:1367] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.008s
I20240502 01:57:11.268075 8479 raft_consensus.cc:348] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:11.268407 8479 raft_consensus.cc:374] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:11.268536 8479 raft_consensus.cc:729] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:11.268946 8479 consensus_queue.cc:260] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:11.269958 8479 ts_tablet_manager.cc:1398] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:57:11.270311 8479 tablet_bootstrap.cc:492] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:11.272709 8479 tablet_bootstrap.cc:654] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:11.273666 8478 tablet_bootstrap.cc:492] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:11.273936 8478 ts_tablet_manager.cc:1367] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.005s
I20240502 01:57:11.274794 8477 tablet_bootstrap.cc:492] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:11.274915 8478 raft_consensus.cc:348] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:11.275058 8477 ts_tablet_manager.cc:1367] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.014s
I20240502 01:57:11.275255 8478 raft_consensus.cc:374] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:11.275400 8478 raft_consensus.cc:729] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:11.275699 8480 tablet_bootstrap.cc:492] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:11.275765 8478 consensus_queue.cc:260] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:11.275939 8480 ts_tablet_manager.cc:1367] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.010s
I20240502 01:57:11.275905 8477 raft_consensus.cc:348] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:11.276202 8477 raft_consensus.cc:374] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:11.276330 8477 raft_consensus.cc:729] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:11.276422 8478 ts_tablet_manager.cc:1398] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:57:11.276624 8477 consensus_queue.cc:260] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:11.277523 8479 tablet_bootstrap.cc:492] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:11.277779 8479 ts_tablet_manager.cc:1367] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240502 01:57:11.277794 8477 ts_tablet_manager.cc:1398] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:57:11.278116 8477 tablet_bootstrap.cc:492] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:11.278764 8479 raft_consensus.cc:348] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:11.278843 8480 raft_consensus.cc:348] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:11.279076 8480 raft_consensus.cc:374] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:11.279237 8479 raft_consensus.cc:374] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:11.279227 8480 raft_consensus.cc:729] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:11.279381 8479 raft_consensus.cc:729] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:11.279541 8480 consensus_queue.cc:260] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:11.279762 8479 consensus_queue.cc:260] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:11.280062 8477 tablet_bootstrap.cc:654] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:11.280234 8480 ts_tablet_manager.cc:1398] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:57:11.280575 8479 ts_tablet_manager.cc:1398] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:57:11.280966 8479 tablet_bootstrap.cc:492] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:11.282691 8479 tablet_bootstrap.cc:654] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:11.286581 8477 tablet_bootstrap.cc:492] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:11.286841 8477 ts_tablet_manager.cc:1367] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240502 01:57:11.287825 8477 raft_consensus.cc:348] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:11.288043 8477 raft_consensus.cc:374] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:11.288049 8479 tablet_bootstrap.cc:492] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:11.288118 8477 raft_consensus.cc:729] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:11.288228 8479 ts_tablet_manager.cc:1367] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 01:57:11.288349 8477 consensus_queue.cc:260] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:11.288915 8477 ts_tablet_manager.cc:1398] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:57:11.289052 8479 raft_consensus.cc:348] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:11.289218 8479 raft_consensus.cc:374] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:11.289296 8479 raft_consensus.cc:729] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:11.289510 8479 consensus_queue.cc:260] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:11.289990 8479 ts_tablet_manager.cc:1398] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:57:11.377278 8398 raft_consensus.cc:482] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:11.377522 8398 raft_consensus.cc:504] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:11.379328 8398 leader_election.cc:288] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:11.387089 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26522a1c09bf44aca91d0410a7459b26" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412" is_pre_election: true
I20240502 01:57:11.387221 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26522a1c09bf44aca91d0410a7459b26" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d" is_pre_election: true
I20240502 01:57:11.387404 7779 raft_consensus.cc:2454] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:11.387490 7516 raft_consensus.cc:2454] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:11.388131 7579 leader_election.cc:302] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:11.388550 8398 raft_consensus.cc:2781] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:11.388697 8398 raft_consensus.cc:482] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:11.388805 8398 raft_consensus.cc:3037] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:11.390949 8398 raft_consensus.cc:504] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:11.391693 8398 leader_election.cc:288] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:11.392274 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26522a1c09bf44aca91d0410a7459b26" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:11.392478 7516 raft_consensus.cc:3037] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:11.394506 7516 raft_consensus.cc:2454] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:11.394966 7581 leader_election.cc:302] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:11.394929 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26522a1c09bf44aca91d0410a7459b26" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:11.395227 7779 raft_consensus.cc:3037] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:11.395273 8398 raft_consensus.cc:2781] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:11.395776 8398 raft_consensus.cc:686] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:11.396333 8398 consensus_queue.cc:237] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:11.397295 7779 raft_consensus.cc:2454] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:11.399704 7232 catalog_manager.cc:5484] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 1 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:11.421272 8397 raft_consensus.cc:482] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:11.421528 8397 raft_consensus.cc:504] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:11.423362 8397 leader_election.cc:288] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:11.438066 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29dce0c6f40a4358ab8855cf0c709862" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5" is_pre_election: true
I20240502 01:57:11.438385 7382 raft_consensus.cc:2454] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 0.
I20240502 01:57:11.439007 7450 leader_election.cc:302] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters:
I20240502 01:57:11.439597 8397 raft_consensus.cc:2781] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:11.439734 8397 raft_consensus.cc:482] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:11.439828 8397 raft_consensus.cc:3037] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:11.442304 8397 raft_consensus.cc:504] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:11.443872 8397 leader_election.cc:288] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:11.444029 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29dce0c6f40a4358ab8855cf0c709862" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:11.444293 7382 raft_consensus.cc:3037] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:11.444836 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29dce0c6f40a4358ab8855cf0c709862" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786" is_pre_election: true
I20240502 01:57:11.444962 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29dce0c6f40a4358ab8855cf0c709862" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:11.445127 7648 raft_consensus.cc:2454] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 0.
I20240502 01:57:11.447374 7382 raft_consensus.cc:2454] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 1.
I20240502 01:57:11.447835 7450 leader_election.cc:302] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters: e8314214a0484251964e4b83a87b6786
I20240502 01:57:11.448348 8397 raft_consensus.cc:2781] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:11.448868 8397 raft_consensus.cc:686] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 1 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:11.449493 8397 consensus_queue.cc:237] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:11.454670 7232 catalog_manager.cc:5484] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 0 to 1, leader changed from <none> to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 1 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
I20240502 01:57:11.622798 8398 raft_consensus.cc:482] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:11.623075 8398 raft_consensus.cc:504] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:11.623791 8398 leader_election.cc:288] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:11.624219 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ee1cc0eacc45149367f418f4e42954" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d" is_pre_election: true
I20240502 01:57:11.624343 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ee1cc0eacc45149367f418f4e42954" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412" is_pre_election: true
I20240502 01:57:11.624440 7516 raft_consensus.cc:2454] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:11.624598 7779 raft_consensus.cc:2454] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:11.624833 7581 leader_election.cc:302] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:11.625176 8398 raft_consensus.cc:2781] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:11.625262 8398 raft_consensus.cc:482] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:11.625372 8398 raft_consensus.cc:3037] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:11.627326 8398 raft_consensus.cc:504] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:11.628050 8398 leader_election.cc:288] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:11.628511 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ee1cc0eacc45149367f418f4e42954" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:11.628628 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ee1cc0eacc45149367f418f4e42954" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:11.628789 7516 raft_consensus.cc:3037] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:11.628865 7779 raft_consensus.cc:3037] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:11.631700 7516 raft_consensus.cc:2454] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:11.631716 7779 raft_consensus.cc:2454] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:11.632246 7579 leader_election.cc:302] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:11.632581 8398 raft_consensus.cc:2781] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:11.632771 8398 raft_consensus.cc:686] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:11.633127 8398 consensus_queue.cc:237] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:11.636006 7232 catalog_manager.cc:5484] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 1 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:11.789568 8398 consensus_queue.cc:1035] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:11.795428 8398 consensus_queue.cc:1035] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:12.035917 7648 raft_consensus.cc:3037] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:12.038915 8397 consensus_queue.cc:1035] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:12.046576 8397 consensus_queue.cc:1035] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:12.050799 8398 consensus_queue.cc:1035] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:12.059898 8398 consensus_queue.cc:1035] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:57:12.068935 8502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:12.069294 8502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:12.077230 8502 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 244.454 ms
time per row: 0.122227 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:57:13.431388 7232 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57734:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:57:13.441661 7358 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9c54c770fd47e4b92a804de3b2bcf3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0f05479dbad84201900bb61c41fc701e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:13.441774 7494 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9c54c770fd47e4b92a804de3b2bcf3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0f05479dbad84201900bb61c41fc701e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:13.442040 7759 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9c54c770fd47e4b92a804de3b2bcf3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0f05479dbad84201900bb61c41fc701e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:13.442201 7494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9c54c770fd47e4b92a804de3b2bcf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:13.442214 7358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9c54c770fd47e4b92a804de3b2bcf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:13.442507 7759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9c54c770fd47e4b92a804de3b2bcf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:13.442456 7361 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6946feb1c14a1eaacad72be52108f3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0f05479dbad84201900bb61c41fc701e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:13.442855 7361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6946feb1c14a1eaacad72be52108f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:13.454437 7627 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6946feb1c14a1eaacad72be52108f3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0f05479dbad84201900bb61c41fc701e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:13.455024 7627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6946feb1c14a1eaacad72be52108f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:13.455245 7758 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6946feb1c14a1eaacad72be52108f3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0f05479dbad84201900bb61c41fc701e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:13.455652 7758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6946feb1c14a1eaacad72be52108f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:13.457332 8570 tablet_bootstrap.cc:492] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:13.459606 8570 tablet_bootstrap.cc:654] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:13.463088 7360 tablet_service.cc:1467] Processing CreateTablet for tablet d3c56e67fc824a8799e0218b245f52c9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0f05479dbad84201900bb61c41fc701e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:13.463770 7360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c56e67fc824a8799e0218b245f52c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:13.464537 8571 tablet_bootstrap.cc:492] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:13.468005 8573 tablet_bootstrap.cc:492] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:13.469353 7493 tablet_service.cc:1467] Processing CreateTablet for tablet d3c56e67fc824a8799e0218b245f52c9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0f05479dbad84201900bb61c41fc701e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:13.470006 7493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c56e67fc824a8799e0218b245f52c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:13.474141 7759 tablet_service.cc:1467] Processing CreateTablet for tablet d3c56e67fc824a8799e0218b245f52c9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0f05479dbad84201900bb61c41fc701e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:13.474749 7759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c56e67fc824a8799e0218b245f52c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:13.478184 8571 tablet_bootstrap.cc:654] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:13.481655 8573 tablet_bootstrap.cc:654] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:13.485033 8571 tablet_bootstrap.cc:492] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:13.485286 8571 ts_tablet_manager.cc:1367] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240502 01:57:13.486128 8571 raft_consensus.cc:348] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:13.486538 8571 raft_consensus.cc:374] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:13.488399 8571 raft_consensus.cc:729] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:13.488811 8571 consensus_queue.cc:260] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:13.489744 8571 ts_tablet_manager.cc:1398] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:57:13.490129 8571 tablet_bootstrap.cc:492] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:13.491256 8570 tablet_bootstrap.cc:492] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:13.491442 8570 ts_tablet_manager.cc:1367] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.020s
I20240502 01:57:13.491843 8571 tablet_bootstrap.cc:654] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:13.492303 8570 raft_consensus.cc:348] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:13.492549 8570 raft_consensus.cc:374] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:13.492676 8570 raft_consensus.cc:729] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:13.493062 8570 consensus_queue.cc:260] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:13.493974 8573 tablet_bootstrap.cc:492] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:13.494195 8573 ts_tablet_manager.cc:1367] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20240502 01:57:13.494438 8570 ts_tablet_manager.cc:1398] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 01:57:13.494813 8570 tablet_bootstrap.cc:492] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:13.495280 8573 raft_consensus.cc:348] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:13.495609 8573 raft_consensus.cc:374] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:13.495765 8573 raft_consensus.cc:729] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:13.496763 8570 tablet_bootstrap.cc:654] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:13.497412 8569 tablet_bootstrap.cc:492] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:13.496846 8573 consensus_queue.cc:260] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:13.498571 8573 ts_tablet_manager.cc:1398] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:57:13.498927 8570 tablet_bootstrap.cc:492] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:13.499156 8570 ts_tablet_manager.cc:1367] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:57:13.499222 8569 tablet_bootstrap.cc:654] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:13.499770 8571 tablet_bootstrap.cc:492] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:13.500022 8571 ts_tablet_manager.cc:1367] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 01:57:13.500065 8570 raft_consensus.cc:348] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:13.500416 8570 raft_consensus.cc:374] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:13.500560 8570 raft_consensus.cc:729] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:13.500962 8570 consensus_queue.cc:260] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:13.501376 8571 raft_consensus.cc:348] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:13.501688 8571 raft_consensus.cc:374] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:13.501816 8571 raft_consensus.cc:729] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:13.501901 8570 ts_tablet_manager.cc:1398] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:57:13.502142 8571 consensus_queue.cc:260] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:13.502382 8570 tablet_bootstrap.cc:492] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:13.503520 8571 ts_tablet_manager.cc:1398] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:57:13.503894 8571 tablet_bootstrap.cc:492] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:13.504664 8570 tablet_bootstrap.cc:654] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:13.505832 8571 tablet_bootstrap.cc:654] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:13.510715 8570 tablet_bootstrap.cc:492] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:13.510776 8571 tablet_bootstrap.cc:492] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:13.510887 8570 ts_tablet_manager.cc:1367] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240502 01:57:13.510958 8571 ts_tablet_manager.cc:1367] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20240502 01:57:13.511886 8571 raft_consensus.cc:348] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:13.511904 8570 raft_consensus.cc:348] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:13.512118 8570 raft_consensus.cc:374] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:13.512188 8571 raft_consensus.cc:374] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:13.512256 8570 raft_consensus.cc:729] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:13.512315 8571 raft_consensus.cc:729] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:13.512521 8569 tablet_bootstrap.cc:492] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:13.512509 8570 consensus_queue.cc:260] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:13.512722 8569 ts_tablet_manager.cc:1367] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.009s
I20240502 01:57:13.512627 8571 consensus_queue.cc:260] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:13.513577 8571 ts_tablet_manager.cc:1398] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:13.513136 8570 ts_tablet_manager.cc:1398] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:57:13.513674 8569 raft_consensus.cc:348] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:13.513887 8569 raft_consensus.cc:374] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:13.513993 8569 raft_consensus.cc:729] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:13.514242 8569 consensus_queue.cc:260] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:13.514775 8569 ts_tablet_manager.cc:1398] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:57:13.515094 8569 tablet_bootstrap.cc:492] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:13.516780 8569 tablet_bootstrap.cc:654] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:13.519233 8569 tablet_bootstrap.cc:492] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:13.519388 8569 ts_tablet_manager.cc:1367] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:57:13.520161 8569 raft_consensus.cc:348] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:13.520351 8569 raft_consensus.cc:374] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:13.520408 8569 raft_consensus.cc:729] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:13.520659 8569 consensus_queue.cc:260] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:13.521190 8569 ts_tablet_manager.cc:1398] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:13.524295 8394 raft_consensus.cc:482] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:13.524473 8394 raft_consensus.cc:504] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:13.525313 8394 leader_election.cc:288] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:13.525817 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c56e67fc824a8799e0218b245f52c9" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5" is_pre_election: true
I20240502 01:57:13.526166 7382 raft_consensus.cc:2454] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 0.
I20240502 01:57:13.526911 7713 leader_election.cc:302] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:13.527210 8394 raft_consensus.cc:2781] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:13.527329 8394 raft_consensus.cc:482] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:13.527449 8394 raft_consensus.cc:3037] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:13.529940 8394 raft_consensus.cc:504] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:13.530789 8394 leader_election.cc:288] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:13.531248 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c56e67fc824a8799e0218b245f52c9" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:13.531620 7382 raft_consensus.cc:3037] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:13.533450 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c56e67fc824a8799e0218b245f52c9" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:13.533393 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c56e67fc824a8799e0218b245f52c9" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d" is_pre_election: true
I20240502 01:57:13.533659 7515 raft_consensus.cc:3037] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:13.534199 7382 raft_consensus.cc:2454] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 1.
I20240502 01:57:13.534729 7713 leader_election.cc:302] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:13.535118 8394 raft_consensus.cc:2781] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:13.535310 8394 raft_consensus.cc:686] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [term 1 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:13.535645 8394 consensus_queue.cc:237] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:13.536535 7515 raft_consensus.cc:2454] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 1.
I20240502 01:57:13.539130 7229 catalog_manager.cc:5484] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 1 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } }
I20240502 01:57:13.635514 8512 raft_consensus.cc:482] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:13.635769 8512 raft_consensus.cc:504] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:13.636657 8512 leader_election.cc:288] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:13.637427 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9c54c770fd47e4b92a804de3b2bcf3" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5" is_pre_election: true
I20240502 01:57:13.637809 7382 raft_consensus.cc:2454] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 0.
I20240502 01:57:13.638366 7450 leader_election.cc:302] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters:
I20240502 01:57:13.638746 8512 raft_consensus.cc:2781] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:13.638900 8512 raft_consensus.cc:482] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:13.639036 8512 raft_consensus.cc:3037] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:13.641883 8512 raft_consensus.cc:504] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:13.642637 8512 leader_election.cc:288] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:13.643244 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9c54c770fd47e4b92a804de3b2bcf3" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:13.643523 7382 raft_consensus.cc:3037] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:13.644906 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9c54c770fd47e4b92a804de3b2bcf3" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412" is_pre_election: true
I20240502 01:57:13.645022 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9c54c770fd47e4b92a804de3b2bcf3" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:13.645176 7779 raft_consensus.cc:2454] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 0.
I20240502 01:57:13.646248 7382 raft_consensus.cc:2454] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 1.
I20240502 01:57:13.646682 7450 leader_election.cc:302] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters: fb68cc03fe5644e0b4846a80d385c412
I20240502 01:57:13.647032 8512 raft_consensus.cc:2781] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:13.647500 8512 raft_consensus.cc:686] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 1 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:13.647828 8512 consensus_queue.cc:237] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:13.650539 7229 catalog_manager.cc:5484] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 0 to 1, leader changed from <none> to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 1 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:13.656303 8517 raft_consensus.cc:482] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:13.656499 8517 raft_consensus.cc:504] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:13.657316 8517 leader_election.cc:288] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:13.657927 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6946feb1c14a1eaacad72be52108f3" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412" is_pre_election: true
I20240502 01:57:13.658277 7780 raft_consensus.cc:2454] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:13.658728 7579 leader_election.cc:302] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:13.659013 8517 raft_consensus.cc:2781] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:13.659139 8517 raft_consensus.cc:482] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:13.659246 8517 raft_consensus.cc:3037] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:13.661350 8517 raft_consensus.cc:504] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:13.661993 8517 leader_election.cc:288] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:13.662547 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6946feb1c14a1eaacad72be52108f3" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:13.662762 7780 raft_consensus.cc:3037] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:13.664145 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6946feb1c14a1eaacad72be52108f3" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5" is_pre_election: true
I20240502 01:57:13.664238 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6946feb1c14a1eaacad72be52108f3" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:13.664417 7382 raft_consensus.cc:2454] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:13.665283 7780 raft_consensus.cc:2454] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:13.665642 7579 leader_election.cc:302] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters: 87e2815274bf41c4ace764c5e37599b5
I20240502 01:57:13.665953 8517 raft_consensus.cc:2781] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:13.666327 8517 raft_consensus.cc:686] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:13.666678 8517 consensus_queue.cc:237] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:13.669499 7229 catalog_manager.cc:5484] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 1 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:14.053833 8394 consensus_queue.cc:1035] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:14.060709 8394 consensus_queue.cc:1035] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:14.067808 7780 raft_consensus.cc:3037] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:14.070540 8512 consensus_queue.cc:1035] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:14.075896 8512 consensus_queue.cc:1035] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:57:14.077829 8595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:14.078361 8595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:14.089202 8595 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:57:14.100278 8517 consensus_queue.cc:1035] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:14.106550 7382 raft_consensus.cc:3037] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:14.110201 8517 consensus_queue.cc:1035] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 252.133 ms
time per row: 0.126067 ms
I20240502 01:57:14.502218 7235 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57734:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:57:14.511924 7494 tablet_service.cc:1467] Processing CreateTablet for tablet 7ac302b18a4a41239b479f539f590b40 (DEFAULT_TABLE table=rebalance_test_table_3 [id=43df8647bdd242a19b1090c7da60f9ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:14.512001 7627 tablet_service.cc:1467] Processing CreateTablet for tablet 7ac302b18a4a41239b479f539f590b40 (DEFAULT_TABLE table=rebalance_test_table_3 [id=43df8647bdd242a19b1090c7da60f9ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:14.512286 7360 tablet_service.cc:1467] Processing CreateTablet for tablet 7ac302b18a4a41239b479f539f590b40 (DEFAULT_TABLE table=rebalance_test_table_3 [id=43df8647bdd242a19b1090c7da60f9ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:14.512459 7494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac302b18a4a41239b479f539f590b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:14.512490 7627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac302b18a4a41239b479f539f590b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:14.512708 7360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac302b18a4a41239b479f539f590b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:14.515079 7628 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca112cbaf214bd7aad56b8e1a336ddd (DEFAULT_TABLE table=rebalance_test_table_3 [id=43df8647bdd242a19b1090c7da60f9ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:14.515476 7628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca112cbaf214bd7aad56b8e1a336ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:14.519711 7626 tablet_service.cc:1467] Processing CreateTablet for tablet 9819568333624685aafc31d82e22f0e0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=43df8647bdd242a19b1090c7da60f9ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:14.519974 7759 tablet_service.cc:1467] Processing CreateTablet for tablet 9819568333624685aafc31d82e22f0e0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=43df8647bdd242a19b1090c7da60f9ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:14.520155 7626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9819568333624685aafc31d82e22f0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:14.520072 7758 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca112cbaf214bd7aad56b8e1a336ddd (DEFAULT_TABLE table=rebalance_test_table_3 [id=43df8647bdd242a19b1090c7da60f9ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:14.520454 7759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9819568333624685aafc31d82e22f0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:14.521683 7758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca112cbaf214bd7aad56b8e1a336ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:14.521617 7361 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca112cbaf214bd7aad56b8e1a336ddd (DEFAULT_TABLE table=rebalance_test_table_3 [id=43df8647bdd242a19b1090c7da60f9ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:14.522049 7361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca112cbaf214bd7aad56b8e1a336ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:14.525825 7493 tablet_service.cc:1467] Processing CreateTablet for tablet 9819568333624685aafc31d82e22f0e0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=43df8647bdd242a19b1090c7da60f9ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:14.526325 7493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9819568333624685aafc31d82e22f0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:14.530786 8657 tablet_bootstrap.cc:492] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:14.532994 8657 tablet_bootstrap.cc:654] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:14.538242 8661 tablet_bootstrap.cc:492] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:14.538658 8659 tablet_bootstrap.cc:492] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:14.539541 8658 tablet_bootstrap.cc:492] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:14.540227 8657 tablet_bootstrap.cc:492] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:14.540405 8657 ts_tablet_manager.cc:1367] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 01:57:14.540712 8659 tablet_bootstrap.cc:654] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:14.541360 8657 raft_consensus.cc:348] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:14.541604 8657 raft_consensus.cc:374] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:14.541704 8657 raft_consensus.cc:729] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:14.541980 8657 consensus_queue.cc:260] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:14.542246 8661 tablet_bootstrap.cc:654] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:14.542382 8658 tablet_bootstrap.cc:654] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:14.543465 8657 ts_tablet_manager.cc:1398] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:57:14.543799 8657 tablet_bootstrap.cc:492] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:14.546329 8657 tablet_bootstrap.cc:654] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:14.549438 8659 tablet_bootstrap.cc:492] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:14.549659 8659 ts_tablet_manager.cc:1367] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240502 01:57:14.550853 8659 raft_consensus.cc:348] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:14.551172 8659 raft_consensus.cc:374] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:14.551276 8659 raft_consensus.cc:729] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:14.551663 8659 consensus_queue.cc:260] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:14.552552 8659 ts_tablet_manager.cc:1398] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:57:14.552902 8659 tablet_bootstrap.cc:492] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:14.554142 8658 tablet_bootstrap.cc:492] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:14.554394 8658 ts_tablet_manager.cc:1367] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20240502 01:57:14.554719 8659 tablet_bootstrap.cc:654] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:14.555852 8658 raft_consensus.cc:348] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:14.556699 8658 raft_consensus.cc:374] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:14.556859 8658 raft_consensus.cc:729] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:14.557202 8659 tablet_bootstrap.cc:492] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:14.557387 8659 ts_tablet_manager.cc:1367] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 01:57:14.557346 8658 consensus_queue.cc:260] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:14.558207 8659 raft_consensus.cc:348] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:14.558434 8659 raft_consensus.cc:374] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:14.558516 8659 raft_consensus.cc:729] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:14.558807 8659 consensus_queue.cc:260] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:14.559531 8659 ts_tablet_manager.cc:1398] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:14.560514 8658 ts_tablet_manager.cc:1398] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 01:57:14.561092 8658 tablet_bootstrap.cc:492] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:14.561650 8661 tablet_bootstrap.cc:492] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:14.561823 8661 ts_tablet_manager.cc:1367] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.007s
I20240502 01:57:14.562489 8661 raft_consensus.cc:348] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:14.562656 8661 raft_consensus.cc:374] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:14.562794 8661 raft_consensus.cc:729] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:14.563027 8658 tablet_bootstrap.cc:654] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:14.563055 8661 consensus_queue.cc:260] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:14.563830 8661 ts_tablet_manager.cc:1398] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:57:14.564147 8661 tablet_bootstrap.cc:492] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:14.565830 8658 tablet_bootstrap.cc:492] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:14.565846 8657 tablet_bootstrap.cc:492] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:14.566009 8658 ts_tablet_manager.cc:1367] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 01:57:14.566015 8657 ts_tablet_manager.cc:1367] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.003s
I20240502 01:57:14.566816 8657 raft_consensus.cc:348] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:14.566815 8658 raft_consensus.cc:348] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:14.567051 8658 raft_consensus.cc:374] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:14.567062 8657 raft_consensus.cc:374] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:14.567130 8658 raft_consensus.cc:729] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:14.567144 8661 tablet_bootstrap.cc:654] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:14.567137 8657 raft_consensus.cc:729] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:14.567404 8657 consensus_queue.cc:260] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:14.567438 8658 consensus_queue.cc:260] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:14.567971 8657 ts_tablet_manager.cc:1398] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:57:14.568061 8658 ts_tablet_manager.cc:1398] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:57:14.568337 8658 tablet_bootstrap.cc:492] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:14.571313 8658 tablet_bootstrap.cc:654] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:14.573364 8661 tablet_bootstrap.cc:492] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:14.573545 8661 ts_tablet_manager.cc:1367] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20240502 01:57:14.574277 8658 tablet_bootstrap.cc:492] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:14.574198 8661 raft_consensus.cc:348] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:14.574396 8661 raft_consensus.cc:374] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:14.574412 8658 ts_tablet_manager.cc:1367] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240502 01:57:14.574494 8661 raft_consensus.cc:729] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:14.574767 8661 consensus_queue.cc:260] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:14.575057 8658 raft_consensus.cc:348] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:14.575263 8661 ts_tablet_manager.cc:1398] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:57:14.575291 8658 raft_consensus.cc:374] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:14.575404 8658 raft_consensus.cc:729] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:14.575640 8658 consensus_queue.cc:260] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:14.576795 8658 ts_tablet_manager.cc:1398] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:57:14.598081 8517 raft_consensus.cc:482] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:14.598340 8517 raft_consensus.cc:504] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:14.599231 8517 leader_election.cc:288] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:14.599861 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac302b18a4a41239b479f539f590b40" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d" is_pre_election: true
I20240502 01:57:14.599891 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac302b18a4a41239b479f539f590b40" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5" is_pre_election: true
I20240502 01:57:14.600160 7515 raft_consensus.cc:2454] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:14.600158 7381 raft_consensus.cc:2454] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:14.600632 7581 leader_election.cc:302] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:14.600955 8517 raft_consensus.cc:2781] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:14.601052 8517 raft_consensus.cc:482] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:14.601161 8517 raft_consensus.cc:3037] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:14.603103 8517 raft_consensus.cc:504] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:14.603698 8517 leader_election.cc:288] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:14.604174 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac302b18a4a41239b479f539f590b40" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:14.604446 7515 raft_consensus.cc:3037] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:14.604389 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac302b18a4a41239b479f539f590b40" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:14.604616 7381 raft_consensus.cc:3037] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:14.606541 7515 raft_consensus.cc:2454] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:14.606631 7381 raft_consensus.cc:2454] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:14.607034 7581 leader_election.cc:302] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:14.607421 8517 raft_consensus.cc:2781] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:14.607966 8517 raft_consensus.cc:686] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:14.608264 8517 consensus_queue.cc:237] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:14.613808 7235 catalog_manager.cc:5484] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 1 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:14.644146 8517 raft_consensus.cc:482] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:14.644366 8517 raft_consensus.cc:504] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:14.645112 8517 leader_election.cc:288] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:14.645785 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca112cbaf214bd7aad56b8e1a336ddd" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5" is_pre_election: true
I20240502 01:57:14.645861 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca112cbaf214bd7aad56b8e1a336ddd" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412" is_pre_election: true
I20240502 01:57:14.646029 7381 raft_consensus.cc:2454] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:14.646265 7779 raft_consensus.cc:2454] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:14.646538 7581 leader_election.cc:302] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:14.646894 8517 raft_consensus.cc:2781] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:14.647009 8517 raft_consensus.cc:482] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:14.647094 8517 raft_consensus.cc:3037] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:14.649170 8517 raft_consensus.cc:504] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:14.649847 8517 leader_election.cc:288] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:14.650501 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca112cbaf214bd7aad56b8e1a336ddd" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:14.650524 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca112cbaf214bd7aad56b8e1a336ddd" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:14.650780 7381 raft_consensus.cc:3037] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:14.650810 7779 raft_consensus.cc:3037] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:14.653892 7779 raft_consensus.cc:2454] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:14.653910 7381 raft_consensus.cc:2454] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:14.654543 7579 leader_election.cc:302] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:14.654917 8517 raft_consensus.cc:2781] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:14.655107 8517 raft_consensus.cc:686] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:14.655380 8517 consensus_queue.cc:237] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:14.658334 7235 catalog_manager.cc:5484] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 1 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:14.718525 8589 raft_consensus.cc:482] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:14.718760 8589 raft_consensus.cc:504] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:14.719521 8589 leader_election.cc:288] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:14.720219 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9819568333624685aafc31d82e22f0e0" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412" is_pre_election: true
I20240502 01:57:14.720232 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9819568333624685aafc31d82e22f0e0" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786" is_pre_election: true
I20240502 01:57:14.720453 7779 raft_consensus.cc:2454] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 0.
I20240502 01:57:14.720551 7648 raft_consensus.cc:2454] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 0.
I20240502 01:57:14.720975 7448 leader_election.cc:302] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:14.721357 8589 raft_consensus.cc:2781] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:14.721482 8589 raft_consensus.cc:482] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:14.721582 8589 raft_consensus.cc:3037] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:14.723992 8589 raft_consensus.cc:504] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:14.724709 8589 leader_election.cc:288] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:14.725188 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9819568333624685aafc31d82e22f0e0" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:14.725217 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9819568333624685aafc31d82e22f0e0" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:14.725390 7779 raft_consensus.cc:3037] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:14.725432 7648 raft_consensus.cc:3037] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:14.727422 7648 raft_consensus.cc:2454] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 1.
I20240502 01:57:14.727437 7779 raft_consensus.cc:2454] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 1.
I20240502 01:57:14.727871 7448 leader_election.cc:302] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:14.728284 8589 raft_consensus.cc:2781] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:14.728451 8589 raft_consensus.cc:686] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 1 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:14.728749 8589 consensus_queue.cc:237] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:14.731870 7235 catalog_manager.cc:5484] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 0 to 1, leader changed from <none> to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 1 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } }
I20240502 01:57:15.005110 8517 consensus_queue.cc:1035] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:15.012153 8671 consensus_queue.cc:1035] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:15.136178 8593 consensus_queue.cc:1035] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:15.141633 8593 consensus_queue.cc:1035] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:57:15.157440 8675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:15.157824 8675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:15.165607 8675 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 01:57:15.231029 8589 consensus_queue.cc:1035] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:15.272660 8589 consensus_queue.cc:1035] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 310.444 ms
time per row: 0.155222 ms
I20240502 01:57:15.630615 7235 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57734:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:57:15.640496 7360 tablet_service.cc:1467] Processing CreateTablet for tablet 04426a27d50548b9a3eb0e3eb09a5bcc (DEFAULT_TABLE table=rebalance_test_table_4 [id=7b508ba3be264f1fa321170f3d8527fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:15.640738 7493 tablet_service.cc:1467] Processing CreateTablet for tablet 04426a27d50548b9a3eb0e3eb09a5bcc (DEFAULT_TABLE table=rebalance_test_table_4 [id=7b508ba3be264f1fa321170f3d8527fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:15.641258 7493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04426a27d50548b9a3eb0e3eb09a5bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:15.641263 7627 tablet_service.cc:1467] Processing CreateTablet for tablet 04426a27d50548b9a3eb0e3eb09a5bcc (DEFAULT_TABLE table=rebalance_test_table_4 [id=7b508ba3be264f1fa321170f3d8527fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:15.641687 7627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04426a27d50548b9a3eb0e3eb09a5bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:15.641752 7360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04426a27d50548b9a3eb0e3eb09a5bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:15.642019 7361 tablet_service.cc:1467] Processing CreateTablet for tablet 3296950eda164749ab23df9d041ff456 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7b508ba3be264f1fa321170f3d8527fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:15.642391 7361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3296950eda164749ab23df9d041ff456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:15.645735 7494 tablet_service.cc:1467] Processing CreateTablet for tablet 3296950eda164749ab23df9d041ff456 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7b508ba3be264f1fa321170f3d8527fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:15.646232 7494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3296950eda164749ab23df9d041ff456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:15.649739 7759 tablet_service.cc:1467] Processing CreateTablet for tablet 3296950eda164749ab23df9d041ff456 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7b508ba3be264f1fa321170f3d8527fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:15.650089 7758 tablet_service.cc:1467] Processing CreateTablet for tablet dffadd26fd38434eaf6cda23da9f1408 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7b508ba3be264f1fa321170f3d8527fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:15.650254 7759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3296950eda164749ab23df9d041ff456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:15.650679 7758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dffadd26fd38434eaf6cda23da9f1408. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:15.651069 7495 tablet_service.cc:1467] Processing CreateTablet for tablet dffadd26fd38434eaf6cda23da9f1408 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7b508ba3be264f1fa321170f3d8527fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:15.651448 7495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dffadd26fd38434eaf6cda23da9f1408. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:15.653625 7628 tablet_service.cc:1467] Processing CreateTablet for tablet dffadd26fd38434eaf6cda23da9f1408 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7b508ba3be264f1fa321170f3d8527fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:15.654223 7628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dffadd26fd38434eaf6cda23da9f1408. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:15.659988 8742 tablet_bootstrap.cc:492] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:15.661877 8741 tablet_bootstrap.cc:492] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:15.662981 8742 tablet_bootstrap.cc:654] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:15.663923 8741 tablet_bootstrap.cc:654] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:15.675374 8740 tablet_bootstrap.cc:492] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:15.675611 8743 tablet_bootstrap.cc:492] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:15.678514 8743 tablet_bootstrap.cc:654] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:15.679960 8740 tablet_bootstrap.cc:654] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:15.681633 8743 tablet_bootstrap.cc:492] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:15.681635 8741 tablet_bootstrap.cc:492] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:15.681865 8741 ts_tablet_manager.cc:1367] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.015s
I20240502 01:57:15.681878 8743 ts_tablet_manager.cc:1367] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 01:57:15.682330 8742 tablet_bootstrap.cc:492] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:15.682665 8742 ts_tablet_manager.cc:1367] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.015s
I20240502 01:57:15.682796 8741 raft_consensus.cc:348] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:15.682796 8743 raft_consensus.cc:348] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:15.683036 8743 raft_consensus.cc:374] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:15.683036 8741 raft_consensus.cc:374] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:15.683140 8741 raft_consensus.cc:729] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:15.683287 8743 raft_consensus.cc:729] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:15.683473 8741 consensus_queue.cc:260] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:15.683647 8743 consensus_queue.cc:260] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:15.684329 8741 ts_tablet_manager.cc:1398] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:57:15.684729 8741 tablet_bootstrap.cc:492] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:15.683813 8742 raft_consensus.cc:348] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:15.685343 8742 raft_consensus.cc:374] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:15.685474 8742 raft_consensus.cc:729] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:15.685868 8742 consensus_queue.cc:260] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:15.687583 8741 tablet_bootstrap.cc:654] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:15.687667 8742 ts_tablet_manager.cc:1398] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:57:15.689110 8742 tablet_bootstrap.cc:492] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:15.690279 8743 ts_tablet_manager.cc:1398] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:57:15.691948 8742 tablet_bootstrap.cc:654] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:15.694845 8742 tablet_bootstrap.cc:492] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:15.694833 8741 tablet_bootstrap.cc:492] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:15.695030 8742 ts_tablet_manager.cc:1367] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 01:57:15.695058 8741 ts_tablet_manager.cc:1367] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20240502 01:57:15.695739 8743 tablet_bootstrap.cc:492] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:15.695864 8742 raft_consensus.cc:348] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:15.695930 8741 raft_consensus.cc:348] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:15.696100 8742 raft_consensus.cc:374] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:15.696172 8742 raft_consensus.cc:729] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:15.696226 8741 raft_consensus.cc:374] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:15.696319 8741 raft_consensus.cc:729] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:15.696435 8742 consensus_queue.cc:260] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:15.696650 8741 consensus_queue.cc:260] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:15.697285 8741 ts_tablet_manager.cc:1398] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:57:15.697898 8743 tablet_bootstrap.cc:654] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:15.698277 8742 ts_tablet_manager.cc:1398] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:57:15.699973 8740 tablet_bootstrap.cc:492] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:15.700997 8740 ts_tablet_manager.cc:1367] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.000s
I20240502 01:57:15.702307 8740 raft_consensus.cc:348] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:15.702741 8740 raft_consensus.cc:374] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:15.702855 8740 raft_consensus.cc:729] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:15.703204 8740 consensus_queue.cc:260] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:15.703650 8743 tablet_bootstrap.cc:492] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:15.703799 8740 ts_tablet_manager.cc:1398] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:57:15.703855 8743 ts_tablet_manager.cc:1367] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20240502 01:57:15.704165 8740 tablet_bootstrap.cc:492] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:15.704718 8743 raft_consensus.cc:348] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:15.704970 8743 raft_consensus.cc:374] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:15.705068 8743 raft_consensus.cc:729] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:15.705333 8743 consensus_queue.cc:260] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:15.705899 8740 tablet_bootstrap.cc:654] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:15.705968 8743 ts_tablet_manager.cc:1398] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 01:57:15.707984 8740 tablet_bootstrap.cc:492] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:15.708153 8740 ts_tablet_manager.cc:1367] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:57:15.709837 8740 raft_consensus.cc:348] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:15.710081 8740 raft_consensus.cc:374] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:15.710175 8740 raft_consensus.cc:729] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:15.710443 8740 consensus_queue.cc:260] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:15.710940 8740 ts_tablet_manager.cc:1398] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:57:15.711306 8740 tablet_bootstrap.cc:492] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:15.712715 8740 tablet_bootstrap.cc:654] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:15.714836 8740 tablet_bootstrap.cc:492] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:15.714990 8740 ts_tablet_manager.cc:1367] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:57:15.715670 8740 raft_consensus.cc:348] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:15.715831 8740 raft_consensus.cc:374] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:15.715898 8740 raft_consensus.cc:729] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:15.716094 8740 consensus_queue.cc:260] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:15.716576 8740 ts_tablet_manager.cc:1398] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:57:15.765496 8671 raft_consensus.cc:482] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:15.765779 8671 raft_consensus.cc:504] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:15.766520 8671 leader_election.cc:288] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:15.767201 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04426a27d50548b9a3eb0e3eb09a5bcc" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5" is_pre_election: true
I20240502 01:57:15.767251 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04426a27d50548b9a3eb0e3eb09a5bcc" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d" is_pre_election: true
I20240502 01:57:15.767443 7381 raft_consensus.cc:2454] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:15.767571 7516 raft_consensus.cc:2454] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:15.767942 7581 leader_election.cc:302] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:15.768267 8671 raft_consensus.cc:2781] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:15.768388 8671 raft_consensus.cc:482] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:15.768503 8671 raft_consensus.cc:3037] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:15.771049 8671 raft_consensus.cc:504] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:15.771679 8671 leader_election.cc:288] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:15.772042 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04426a27d50548b9a3eb0e3eb09a5bcc" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:15.772256 7381 raft_consensus.cc:3037] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:15.772229 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04426a27d50548b9a3eb0e3eb09a5bcc" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:15.772466 7516 raft_consensus.cc:3037] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:15.774315 7381 raft_consensus.cc:2454] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:15.774806 7581 leader_election.cc:302] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:15.775125 8671 raft_consensus.cc:2781] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:15.775207 7516 raft_consensus.cc:2454] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:15.775331 8671 raft_consensus.cc:686] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:15.775701 8671 consensus_queue.cc:237] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:15.778779 7235 catalog_manager.cc:5484] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 1 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
I20240502 01:57:15.863842 8671 raft_consensus.cc:482] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:15.864053 8671 raft_consensus.cc:504] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:15.864748 8671 leader_election.cc:288] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:15.865316 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffadd26fd38434eaf6cda23da9f1408" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412" is_pre_election: true
I20240502 01:57:15.865417 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffadd26fd38434eaf6cda23da9f1408" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d" is_pre_election: true
I20240502 01:57:15.865585 7780 raft_consensus.cc:2454] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:15.865686 7516 raft_consensus.cc:2454] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:15.866076 7579 leader_election.cc:302] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:15.866346 8671 raft_consensus.cc:2781] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:15.866475 8671 raft_consensus.cc:482] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:15.866559 8671 raft_consensus.cc:3037] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:15.868593 8671 raft_consensus.cc:504] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:15.869297 8671 leader_election.cc:288] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:15.869863 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffadd26fd38434eaf6cda23da9f1408" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:15.869877 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffadd26fd38434eaf6cda23da9f1408" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:15.870090 7780 raft_consensus.cc:3037] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:15.870091 7516 raft_consensus.cc:3037] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:15.872213 7516 raft_consensus.cc:2454] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:15.872298 7780 raft_consensus.cc:2454] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:15.872681 7581 leader_election.cc:302] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:15.873049 8671 raft_consensus.cc:2781] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:15.873193 8671 raft_consensus.cc:686] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:15.873473 8671 consensus_queue.cc:237] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:15.876436 7235 catalog_manager.cc:5484] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 1 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } }
I20240502 01:57:16.033286 8470 raft_consensus.cc:482] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:16.033515 8470 raft_consensus.cc:504] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:16.034202 8470 leader_election.cc:288] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:16.034837 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296950eda164749ab23df9d041ff456" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d" is_pre_election: true
I20240502 01:57:16.034881 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296950eda164749ab23df9d041ff456" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412" is_pre_election: true
I20240502 01:57:16.035130 7780 raft_consensus.cc:2454] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 0.
I20240502 01:57:16.035132 7516 raft_consensus.cc:2454] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 0.
I20240502 01:57:16.035614 7311 leader_election.cc:302] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:16.035930 8470 raft_consensus.cc:2781] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:16.036065 8470 raft_consensus.cc:482] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:16.036166 8470 raft_consensus.cc:3037] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:16.038512 8470 raft_consensus.cc:504] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:16.039268 8470 leader_election.cc:288] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:16.039917 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296950eda164749ab23df9d041ff456" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:16.040102 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296950eda164749ab23df9d041ff456" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:16.040132 7780 raft_consensus.cc:3037] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:16.040356 7516 raft_consensus.cc:3037] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:16.042492 7516 raft_consensus.cc:2454] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 1.
I20240502 01:57:16.042508 7780 raft_consensus.cc:2454] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 1.
I20240502 01:57:16.042987 7311 leader_election.cc:302] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:16.043393 8470 raft_consensus.cc:2781] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:16.043771 8470 raft_consensus.cc:686] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:16.044088 8470 consensus_queue.cc:237] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:16.047173 7235 catalog_manager.cc:5484] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:16.209287 8671 consensus_queue.cc:1035] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:16.215703 8671 consensus_queue.cc:1035] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:16.300789 8671 consensus_queue.cc:1035] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:16.306507 8764 consensus_queue.cc:1035] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:16.477777 8470 consensus_queue.cc:1035] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:16.482894 8470 consensus_queue.cc:1035] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:57:16.508059 8757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:16.508414 8757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:16.516651 8757 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 268.431 ms
time per row: 0.134216 ms
I20240502 01:57:16.955709 7228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57734:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:57:16.967451 7759 tablet_service.cc:1467] Processing CreateTablet for tablet d3ac865cf3c044ac8ae3b0383b0b73c0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=42bd284155bd4351ad99fbbe71b17ffa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:16.968012 7759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ac865cf3c044ac8ae3b0383b0b73c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:16.968600 7361 tablet_service.cc:1467] Processing CreateTablet for tablet d3ac865cf3c044ac8ae3b0383b0b73c0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=42bd284155bd4351ad99fbbe71b17ffa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:16.968578 7493 tablet_service.cc:1467] Processing CreateTablet for tablet d727fc7ec2054b46aafb44ccf4bd0ad8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=42bd284155bd4351ad99fbbe71b17ffa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:16.968628 7358 tablet_service.cc:1467] Processing CreateTablet for tablet d727fc7ec2054b46aafb44ccf4bd0ad8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=42bd284155bd4351ad99fbbe71b17ffa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:16.969076 7493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d727fc7ec2054b46aafb44ccf4bd0ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:16.969125 7358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d727fc7ec2054b46aafb44ccf4bd0ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:16.969331 7360 tablet_service.cc:1467] Processing CreateTablet for tablet 46549b89a28243a7b4245c14ecba6008 (DEFAULT_TABLE table=rebalance_test_table_5 [id=42bd284155bd4351ad99fbbe71b17ffa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:16.969761 7360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46549b89a28243a7b4245c14ecba6008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:16.975294 7758 tablet_service.cc:1467] Processing CreateTablet for tablet 46549b89a28243a7b4245c14ecba6008 (DEFAULT_TABLE table=rebalance_test_table_5 [id=42bd284155bd4351ad99fbbe71b17ffa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:16.975762 7758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46549b89a28243a7b4245c14ecba6008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:16.979858 8813 tablet_bootstrap.cc:492] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:16.981638 7361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ac865cf3c044ac8ae3b0383b0b73c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:16.982208 7628 tablet_service.cc:1467] Processing CreateTablet for tablet d3ac865cf3c044ac8ae3b0383b0b73c0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=42bd284155bd4351ad99fbbe71b17ffa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:16.982689 8813 tablet_bootstrap.cc:654] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:16.982777 7628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ac865cf3c044ac8ae3b0383b0b73c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:16.985353 7627 tablet_service.cc:1467] Processing CreateTablet for tablet 46549b89a28243a7b4245c14ecba6008 (DEFAULT_TABLE table=rebalance_test_table_5 [id=42bd284155bd4351ad99fbbe71b17ffa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:16.985930 7627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46549b89a28243a7b4245c14ecba6008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:16.989487 7626 tablet_service.cc:1467] Processing CreateTablet for tablet d727fc7ec2054b46aafb44ccf4bd0ad8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=42bd284155bd4351ad99fbbe71b17ffa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:16.990079 7626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d727fc7ec2054b46aafb44ccf4bd0ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:16.995205 8814 tablet_bootstrap.cc:492] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:16.997840 8814 tablet_bootstrap.cc:654] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:16.999727 8815 tablet_bootstrap.cc:492] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:17.002112 8815 tablet_bootstrap.cc:654] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:17.005254 8815 tablet_bootstrap.cc:492] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:17.005506 8815 ts_tablet_manager.cc:1367] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 01:57:17.006181 8814 tablet_bootstrap.cc:492] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:17.006461 8814 ts_tablet_manager.cc:1367] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 01:57:17.006457 8815 raft_consensus.cc:348] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:17.006884 8815 raft_consensus.cc:374] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:17.007035 8815 raft_consensus.cc:729] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:17.007346 8814 raft_consensus.cc:348] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:17.007573 8814 raft_consensus.cc:374] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:17.007660 8814 raft_consensus.cc:729] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:17.007386 8815 consensus_queue.cc:260] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:17.008277 8815 ts_tablet_manager.cc:1398] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:57:17.007902 8814 consensus_queue.cc:260] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:17.011118 8814 ts_tablet_manager.cc:1398] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:57:17.011513 8814 tablet_bootstrap.cc:492] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:17.014968 8813 tablet_bootstrap.cc:492] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:17.015203 8813 ts_tablet_manager.cc:1367] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.013s
I20240502 01:57:17.016165 8813 raft_consensus.cc:348] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:17.016355 8813 raft_consensus.cc:374] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:17.016474 8813 raft_consensus.cc:729] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:17.016810 8813 consensus_queue.cc:260] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:17.017122 8814 tablet_bootstrap.cc:654] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:17.017585 8813 ts_tablet_manager.cc:1398] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:17.018373 8813 tablet_bootstrap.cc:492] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:17.018529 8819 tablet_bootstrap.cc:492] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:17.020267 8813 tablet_bootstrap.cc:654] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:17.021003 8819 tablet_bootstrap.cc:654] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:17.024438 8819 tablet_bootstrap.cc:492] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:17.024755 8819 ts_tablet_manager.cc:1367] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 01:57:17.026023 8819 raft_consensus.cc:348] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:17.027288 8819 raft_consensus.cc:374] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:17.027454 8819 raft_consensus.cc:729] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:17.027815 8813 tablet_bootstrap.cc:492] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:17.027760 8819 consensus_queue.cc:260] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:17.028050 8813 ts_tablet_manager.cc:1367] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20240502 01:57:17.028457 8819 ts_tablet_manager.cc:1398] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:57:17.028945 8819 tablet_bootstrap.cc:492] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:17.028982 8814 tablet_bootstrap.cc:492] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:17.029219 8814 ts_tablet_manager.cc:1367] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.005s
I20240502 01:57:17.030155 8814 raft_consensus.cc:348] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:17.030310 8813 raft_consensus.cc:348] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:17.030459 8814 raft_consensus.cc:374] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:17.030588 8813 raft_consensus.cc:374] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:17.030583 8814 raft_consensus.cc:729] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:17.030692 8813 raft_consensus.cc:729] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:17.030716 8819 tablet_bootstrap.cc:654] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:17.030905 8814 consensus_queue.cc:260] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:17.031023 8813 consensus_queue.cc:260] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:17.031216 8755 raft_consensus.cc:482] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:17.031394 8755 raft_consensus.cc:504] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:17.031720 8813 ts_tablet_manager.cc:1398] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:57:17.031981 8814 ts_tablet_manager.cc:1398] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:57:17.032234 8814 tablet_bootstrap.cc:492] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:17.032528 8755 leader_election.cc:288] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:17.033437 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ac865cf3c044ac8ae3b0383b0b73c0" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412" is_pre_election: true
I20240502 01:57:17.033440 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ac865cf3c044ac8ae3b0383b0b73c0" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786" is_pre_election: true
I20240502 01:57:17.033708 7780 raft_consensus.cc:2454] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 0.
W20240502 01:57:17.034561 7314 leader_election.cc:341] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:57:17.035014 7311 leader_election.cc:302] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters: e8314214a0484251964e4b83a87b6786
I20240502 01:57:17.035202 8814 tablet_bootstrap.cc:654] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:17.036212 8593 raft_consensus.cc:482] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:17.036433 8593 raft_consensus.cc:504] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:17.037472 8593 leader_election.cc:288] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:17.038174 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d727fc7ec2054b46aafb44ccf4bd0ad8" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5" is_pre_election: true
I20240502 01:57:17.038455 7381 raft_consensus.cc:2454] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:17.039057 8819 tablet_bootstrap.cc:492] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:17.038960 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d727fc7ec2054b46aafb44ccf4bd0ad8" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d" is_pre_election: true
I20240502 01:57:17.039216 7516 raft_consensus.cc:2454] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:17.039273 8819 ts_tablet_manager.cc:1367] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 01:57:17.039364 8755 raft_consensus.cc:2781] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:17.039485 8755 raft_consensus.cc:482] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:17.039587 8755 raft_consensus.cc:3037] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:17.040202 8819 raft_consensus.cc:348] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:17.040534 8819 raft_consensus.cc:374] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:17.040645 8819 raft_consensus.cc:729] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:17.040953 8819 consensus_queue.cc:260] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:17.040482 7581 leader_election.cc:302] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:17.041570 8819 ts_tablet_manager.cc:1398] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:17.041914 8819 tablet_bootstrap.cc:492] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:17.041986 8764 raft_consensus.cc:2781] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:17.042147 8764 raft_consensus.cc:482] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:17.042286 8764 raft_consensus.cc:3037] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:17.043802 8819 tablet_bootstrap.cc:654] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:17.044387 8755 raft_consensus.cc:504] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:17.045464 8764 raft_consensus.cc:504] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:17.045750 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ac865cf3c044ac8ae3b0383b0b73c0" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:17.046059 7780 raft_consensus.cc:3037] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:17.048467 8764 leader_election.cc:288] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:17.048980 7780 raft_consensus.cc:2454] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 1.
I20240502 01:57:17.049939 7311 leader_election.cc:302] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:17.050089 8755 leader_election.cc:288] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:17.050493 8470 raft_consensus.cc:2781] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:17.050470 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d727fc7ec2054b46aafb44ccf4bd0ad8" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:17.050720 7516 raft_consensus.cc:3037] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:17.050690 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d727fc7ec2054b46aafb44ccf4bd0ad8" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:17.050920 7381 raft_consensus.cc:3037] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:17.050760 8470 raft_consensus.cc:686] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:17.053719 8470 consensus_queue.cc:237] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:17.054250 7516 raft_consensus.cc:2454] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:17.051066 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ac865cf3c044ac8ae3b0383b0b73c0" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:17.054718 7648 raft_consensus.cc:3037] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:17.055687 7581 leader_election.cc:302] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:17.056109 8593 raft_consensus.cc:2781] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:17.056380 8593 raft_consensus.cc:686] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:17.056993 7381 raft_consensus.cc:2454] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:17.057821 7648 raft_consensus.cc:2454] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 1.
I20240502 01:57:17.056816 8593 consensus_queue.cc:237] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:17.060143 8819 tablet_bootstrap.cc:492] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:17.060349 8819 ts_tablet_manager.cc:1367] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20240502 01:57:17.060493 7235 catalog_manager.cc:5484] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
I20240502 01:57:17.061348 8819 raft_consensus.cc:348] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:17.061563 8819 raft_consensus.cc:374] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:17.061664 8819 raft_consensus.cc:729] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:17.061944 8814 tablet_bootstrap.cc:492] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:17.061882 8819 consensus_queue.cc:260] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:17.062078 8814 ts_tablet_manager.cc:1367] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.002s
I20240502 01:57:17.062494 8819 ts_tablet_manager.cc:1398] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:57:17.062723 8814 raft_consensus.cc:348] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:17.062934 8814 raft_consensus.cc:374] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:17.063050 8814 raft_consensus.cc:729] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:17.062808 7228 catalog_manager.cc:5484] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 1 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
I20240502 01:57:17.063338 8814 consensus_queue.cc:260] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:17.063966 8814 ts_tablet_manager.cc:1398] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:57:17.337446 8593 raft_consensus.cc:482] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:17.337678 8593 raft_consensus.cc:504] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:17.338419 8593 leader_election.cc:288] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:17.339071 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46549b89a28243a7b4245c14ecba6008" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412" is_pre_election: true
I20240502 01:57:17.339126 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46549b89a28243a7b4245c14ecba6008" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5" is_pre_election: true
I20240502 01:57:17.339362 7780 raft_consensus.cc:2454] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:17.339643 7381 raft_consensus.cc:2454] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:17.339897 7579 leader_election.cc:302] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:17.340179 8593 raft_consensus.cc:2781] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:17.340276 8593 raft_consensus.cc:482] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:17.340384 8593 raft_consensus.cc:3037] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:17.342298 8593 raft_consensus.cc:504] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:17.342932 8593 leader_election.cc:288] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:17.343461 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46549b89a28243a7b4245c14ecba6008" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:17.343508 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46549b89a28243a7b4245c14ecba6008" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:17.343688 7780 raft_consensus.cc:3037] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:17.343739 7381 raft_consensus.cc:3037] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:17.345690 7780 raft_consensus.cc:2454] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:17.345883 7381 raft_consensus.cc:2454] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:17.346148 7579 leader_election.cc:302] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:17.346513 8593 raft_consensus.cc:2781] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:17.346647 8593 raft_consensus.cc:686] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:17.346947 8593 consensus_queue.cc:237] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:17.350018 7235 catalog_manager.cc:5484] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 1 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:17.457238 8764 consensus_queue.cc:1035] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:17.473232 8593 consensus_queue.cc:1035] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:17.613606 8755 consensus_queue.cc:1035] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:17.619463 8836 consensus_queue.cc:1035] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:17.725006 8593 consensus_queue.cc:1035] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:17.730123 8807 consensus_queue.cc:1035] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:57:17.766911 8830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:17.767298 8830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:17.775285 8830 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
INSERT report
rows total: 2000
time total: 246.245 ms
time per row: 0.123123 ms
I20240502 01:57:18.181394 7229 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57734:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:57:18.191706 7758 tablet_service.cc:1467] Processing CreateTablet for tablet d5bcb764b10d43419e58fdcfeadb2143 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2cc1a26f208d47cd93b1b43b5169785a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:18.192245 7758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5bcb764b10d43419e58fdcfeadb2143. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:18.192207 7358 tablet_service.cc:1467] Processing CreateTablet for tablet d5bcb764b10d43419e58fdcfeadb2143 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2cc1a26f208d47cd93b1b43b5169785a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:18.192668 7358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5bcb764b10d43419e58fdcfeadb2143. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:18.192647 7759 tablet_service.cc:1467] Processing CreateTablet for tablet 0718691ffdc04f719ad7df57987df68c (DEFAULT_TABLE table=rebalance_test_table_6 [id=2cc1a26f208d47cd93b1b43b5169785a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:18.193094 7757 tablet_service.cc:1467] Processing CreateTablet for tablet 7a14895a05814947bcec5111f41fdfb9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2cc1a26f208d47cd93b1b43b5169785a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:18.193099 7759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0718691ffdc04f719ad7df57987df68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:18.194124 7757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a14895a05814947bcec5111f41fdfb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:18.194690 7360 tablet_service.cc:1467] Processing CreateTablet for tablet 7a14895a05814947bcec5111f41fdfb9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2cc1a26f208d47cd93b1b43b5169785a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:18.195104 7360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a14895a05814947bcec5111f41fdfb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:18.195160 7494 tablet_service.cc:1467] Processing CreateTablet for tablet 7a14895a05814947bcec5111f41fdfb9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2cc1a26f208d47cd93b1b43b5169785a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:18.195593 7494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a14895a05814947bcec5111f41fdfb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:18.196664 7493 tablet_service.cc:1467] Processing CreateTablet for tablet d5bcb764b10d43419e58fdcfeadb2143 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2cc1a26f208d47cd93b1b43b5169785a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:18.197141 7493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5bcb764b10d43419e58fdcfeadb2143. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:18.197360 7495 tablet_service.cc:1467] Processing CreateTablet for tablet 0718691ffdc04f719ad7df57987df68c (DEFAULT_TABLE table=rebalance_test_table_6 [id=2cc1a26f208d47cd93b1b43b5169785a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:18.197777 7495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0718691ffdc04f719ad7df57987df68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:18.201404 7626 tablet_service.cc:1467] Processing CreateTablet for tablet 0718691ffdc04f719ad7df57987df68c (DEFAULT_TABLE table=rebalance_test_table_6 [id=2cc1a26f208d47cd93b1b43b5169785a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:18.201874 7626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0718691ffdc04f719ad7df57987df68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:18.212328 8893 tablet_bootstrap.cc:492] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:18.214385 8893 tablet_bootstrap.cc:654] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:18.219945 8893 tablet_bootstrap.cc:492] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:18.220129 8893 ts_tablet_manager.cc:1367] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 01:57:18.220137 8894 tablet_bootstrap.cc:492] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:18.221755 8893 raft_consensus.cc:348] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:18.222008 8893 raft_consensus.cc:374] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:18.223049 8894 tablet_bootstrap.cc:654] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:18.223464 8893 raft_consensus.cc:729] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:18.223753 8893 consensus_queue.cc:260] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:18.224915 8893 ts_tablet_manager.cc:1398] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:57:18.225795 8893 tablet_bootstrap.cc:492] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:18.227928 8893 tablet_bootstrap.cc:654] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:18.231856 8897 tablet_bootstrap.cc:492] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:18.232394 8891 tablet_bootstrap.cc:492] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:18.233973 8894 tablet_bootstrap.cc:492] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:18.234340 8894 ts_tablet_manager.cc:1367] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20240502 01:57:18.235435 8897 tablet_bootstrap.cc:654] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:18.235395 8894 raft_consensus.cc:348] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:18.235658 8894 raft_consensus.cc:374] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:18.235770 8894 raft_consensus.cc:729] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:18.236078 8894 consensus_queue.cc:260] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:18.236253 8891 tablet_bootstrap.cc:654] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:18.237906 8897 tablet_bootstrap.cc:492] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:18.238066 8897 ts_tablet_manager.cc:1367] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240502 01:57:18.238730 8894 ts_tablet_manager.cc:1398] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:57:18.238934 8897 raft_consensus.cc:348] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:18.239156 8897 raft_consensus.cc:374] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:18.239166 8894 tablet_bootstrap.cc:492] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:18.239243 8897 raft_consensus.cc:729] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:18.239542 8897 consensus_queue.cc:260] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:18.240151 8897 ts_tablet_manager.cc:1398] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:57:18.241448 8894 tablet_bootstrap.cc:654] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:18.242389 8891 tablet_bootstrap.cc:492] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:18.242637 8891 ts_tablet_manager.cc:1367] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 01:57:18.243820 8891 raft_consensus.cc:348] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:18.244009 8891 raft_consensus.cc:374] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:18.244123 8891 raft_consensus.cc:729] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:18.244379 8891 consensus_queue.cc:260] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:18.245076 8891 ts_tablet_manager.cc:1398] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:57:18.245438 8891 tablet_bootstrap.cc:492] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:18.246049 8893 tablet_bootstrap.cc:492] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:18.246356 8893 ts_tablet_manager.cc:1367] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.004s
I20240502 01:57:18.247246 8891 tablet_bootstrap.cc:654] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:18.247644 8893 raft_consensus.cc:348] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:18.248692 8893 raft_consensus.cc:374] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:18.248862 8893 raft_consensus.cc:729] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:18.249329 8893 consensus_queue.cc:260] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:18.251241 8894 tablet_bootstrap.cc:492] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:18.251437 8894 ts_tablet_manager.cc:1367] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20240502 01:57:18.252101 8893 ts_tablet_manager.cc:1398] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240502 01:57:18.252338 8894 raft_consensus.cc:348] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:18.252506 8893 tablet_bootstrap.cc:492] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:18.252612 8894 raft_consensus.cc:374] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:18.252697 8894 raft_consensus.cc:729] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:18.253038 8894 consensus_queue.cc:260] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:18.254546 8894 ts_tablet_manager.cc:1398] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:57:18.254886 8894 tablet_bootstrap.cc:492] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:18.254932 8893 tablet_bootstrap.cc:654] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:18.256909 8894 tablet_bootstrap.cc:654] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:18.261344 8891 tablet_bootstrap.cc:492] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:18.261585 8891 ts_tablet_manager.cc:1367] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20240502 01:57:18.262552 8891 raft_consensus.cc:348] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:18.262830 8891 raft_consensus.cc:374] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:18.262924 8891 raft_consensus.cc:729] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:18.263206 8891 consensus_queue.cc:260] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:18.263825 8891 ts_tablet_manager.cc:1398] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:57:18.265241 8894 tablet_bootstrap.cc:492] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:18.265434 8894 ts_tablet_manager.cc:1367] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20240502 01:57:18.266557 8894 raft_consensus.cc:348] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:18.266844 8894 raft_consensus.cc:374] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:18.266960 8894 raft_consensus.cc:729] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:18.267239 8894 consensus_queue.cc:260] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:18.267495 8893 tablet_bootstrap.cc:492] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:18.267665 8893 ts_tablet_manager.cc:1367] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20240502 01:57:18.267881 8894 ts_tablet_manager.cc:1398] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:57:18.268548 8893 raft_consensus.cc:348] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:18.268792 8893 raft_consensus.cc:374] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:18.268927 8893 raft_consensus.cc:729] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:18.269199 8893 consensus_queue.cc:260] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:18.270735 8893 ts_tablet_manager.cc:1398] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 01:57:18.356088 8755 raft_consensus.cc:482] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:18.356354 8755 raft_consensus.cc:504] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:18.357048 8755 leader_election.cc:288] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:18.357715 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a14895a05814947bcec5111f41fdfb9" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d" is_pre_election: true
I20240502 01:57:18.357688 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a14895a05814947bcec5111f41fdfb9" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412" is_pre_election: true
I20240502 01:57:18.357896 8394 raft_consensus.cc:482] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:18.357997 7516 raft_consensus.cc:2454] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 0.
I20240502 01:57:18.358006 7779 raft_consensus.cc:2454] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 0.
I20240502 01:57:18.358112 8394 raft_consensus.cc:504] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:18.358542 7313 leader_election.cc:302] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters:
I20240502 01:57:18.358857 8394 leader_election.cc:288] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:18.358908 8755 raft_consensus.cc:2781] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:18.359017 8755 raft_consensus.cc:482] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:18.359127 8755 raft_consensus.cc:3037] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:18.359356 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcb764b10d43419e58fdcfeadb2143" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d" is_pre_election: true
I20240502 01:57:18.359616 7516 raft_consensus.cc:2454] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 0.
I20240502 01:57:18.360092 7713 leader_election.cc:302] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:18.360318 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcb764b10d43419e58fdcfeadb2143" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5" is_pre_election: true
I20240502 01:57:18.360617 7379 raft_consensus.cc:2454] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 0.
I20240502 01:57:18.361016 8394 raft_consensus.cc:2781] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:18.361129 8394 raft_consensus.cc:482] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:18.361243 8394 raft_consensus.cc:3037] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:18.362305 8755 raft_consensus.cc:504] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:18.362844 8755 leader_election.cc:288] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:18.363368 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a14895a05814947bcec5111f41fdfb9" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:18.363425 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a14895a05814947bcec5111f41fdfb9" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:18.363556 7779 raft_consensus.cc:3037] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:18.363662 7516 raft_consensus.cc:3037] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:18.363690 8394 raft_consensus.cc:504] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:18.364909 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcb764b10d43419e58fdcfeadb2143" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:18.365121 7515 raft_consensus.cc:3037] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:18.366261 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcb764b10d43419e58fdcfeadb2143" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:18.366473 7379 raft_consensus.cc:3037] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:18.367161 7516 raft_consensus.cc:2454] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 1.
I20240502 01:57:18.367563 7313 leader_election.cc:302] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters:
I20240502 01:57:18.367863 8755 raft_consensus.cc:2781] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:18.367934 7515 raft_consensus.cc:2454] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 1.
I20240502 01:57:18.368048 8755 raft_consensus.cc:686] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:18.368348 8755 consensus_queue.cc:237] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:18.364451 8394 leader_election.cc:288] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:18.369259 7379 raft_consensus.cc:2454] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 1.
I20240502 01:57:18.365851 7779 raft_consensus.cc:2454] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 1.
I20240502 01:57:18.368337 7713 leader_election.cc:302] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:18.370877 8394 raft_consensus.cc:2781] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:18.371522 8394 raft_consensus.cc:686] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [term 1 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:18.371826 8394 consensus_queue.cc:237] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:18.373214 7235 catalog_manager.cc:5484] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:18.374970 7233 catalog_manager.cc:5484] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 1 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:18.388173 8885 raft_consensus.cc:482] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:18.388427 8885 raft_consensus.cc:504] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:18.389429 8885 leader_election.cc:288] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:18.390002 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0718691ffdc04f719ad7df57987df68c" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412" is_pre_election: true
I20240502 01:57:18.390060 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0718691ffdc04f719ad7df57987df68c" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d" is_pre_election: true
I20240502 01:57:18.390275 7779 raft_consensus.cc:2454] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:18.390303 7515 raft_consensus.cc:2454] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:18.390789 7581 leader_election.cc:302] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:18.391104 8885 raft_consensus.cc:2781] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:18.391209 8885 raft_consensus.cc:482] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:18.391276 8885 raft_consensus.cc:3037] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:18.393322 8885 raft_consensus.cc:504] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:18.393954 8885 leader_election.cc:288] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:18.394502 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0718691ffdc04f719ad7df57987df68c" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:18.394562 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0718691ffdc04f719ad7df57987df68c" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:18.394719 7779 raft_consensus.cc:3037] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:18.394827 7515 raft_consensus.cc:3037] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:18.397408 7515 raft_consensus.cc:2454] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:18.397594 7779 raft_consensus.cc:2454] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:18.397855 7581 leader_election.cc:302] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:18.398172 8885 raft_consensus.cc:2781] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:18.398334 8885 raft_consensus.cc:686] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:18.398614 8885 consensus_queue.cc:237] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:18.401769 7233 catalog_manager.cc:5484] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 1 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
W20240502 01:57:18.803601 8910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:18.803983 8910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:18.812093 8910 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:57:18.834630 8593 consensus_queue.cc:1035] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:18.842316 8593 consensus_queue.cc:1035] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240502 01:57:18.898077 8755 consensus_queue.cc:1035] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:18.923164 8943 consensus_queue.cc:1035] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:18.980984 8394 consensus_queue.cc:1035] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:18.995689 8394 consensus_queue.cc:1035] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 291.626 ms
time per row: 0.145813 ms
I20240502 01:57:19.264581 7229 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57734:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:57:19.274784 7626 tablet_service.cc:1467] Processing CreateTablet for tablet 4112f382c8e24680ae2350d9d5d8af54 (DEFAULT_TABLE table=rebalance_test_table_7 [id=83b264553c7b4d8d96c0e7d539019366]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:19.274999 7358 tablet_service.cc:1467] Processing CreateTablet for tablet 4112f382c8e24680ae2350d9d5d8af54 (DEFAULT_TABLE table=rebalance_test_table_7 [id=83b264553c7b4d8d96c0e7d539019366]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:19.275338 7626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4112f382c8e24680ae2350d9d5d8af54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:19.275473 7358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4112f382c8e24680ae2350d9d5d8af54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:19.275660 7759 tablet_service.cc:1467] Processing CreateTablet for tablet 15e07df96bd94ef8b505a7af2ac88db4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=83b264553c7b4d8d96c0e7d539019366]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:19.276085 7759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15e07df96bd94ef8b505a7af2ac88db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:19.280303 8979 tablet_bootstrap.cc:492] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:19.280392 7495 tablet_service.cc:1467] Processing CreateTablet for tablet 4112f382c8e24680ae2350d9d5d8af54 (DEFAULT_TABLE table=rebalance_test_table_7 [id=83b264553c7b4d8d96c0e7d539019366]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:19.282227 7360 tablet_service.cc:1467] Processing CreateTablet for tablet 9c93a84c310742b999bbb053d687639f (DEFAULT_TABLE table=rebalance_test_table_7 [id=83b264553c7b4d8d96c0e7d539019366]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:19.282670 7360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c93a84c310742b999bbb053d687639f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:19.282560 7758 tablet_service.cc:1467] Processing CreateTablet for tablet 9c93a84c310742b999bbb053d687639f (DEFAULT_TABLE table=rebalance_test_table_7 [id=83b264553c7b4d8d96c0e7d539019366]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:19.282897 8979 tablet_bootstrap.cc:654] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:19.283059 7758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c93a84c310742b999bbb053d687639f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:19.280869 7495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4112f382c8e24680ae2350d9d5d8af54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:19.285836 7627 tablet_service.cc:1467] Processing CreateTablet for tablet 15e07df96bd94ef8b505a7af2ac88db4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=83b264553c7b4d8d96c0e7d539019366]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:19.286263 7627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15e07df96bd94ef8b505a7af2ac88db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:19.282122 7494 tablet_service.cc:1467] Processing CreateTablet for tablet 15e07df96bd94ef8b505a7af2ac88db4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=83b264553c7b4d8d96c0e7d539019366]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:19.289633 7494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15e07df96bd94ef8b505a7af2ac88db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:19.293330 7628 tablet_service.cc:1467] Processing CreateTablet for tablet 9c93a84c310742b999bbb053d687639f (DEFAULT_TABLE table=rebalance_test_table_7 [id=83b264553c7b4d8d96c0e7d539019366]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:19.293908 7628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c93a84c310742b999bbb053d687639f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:19.303546 8982 tablet_bootstrap.cc:492] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:19.304796 8980 tablet_bootstrap.cc:492] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:19.306996 8982 tablet_bootstrap.cc:654] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:19.308341 8980 tablet_bootstrap.cc:654] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:19.309764 8984 tablet_bootstrap.cc:492] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:19.311771 8984 tablet_bootstrap.cc:654] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:19.315479 8982 tablet_bootstrap.cc:492] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:19.315523 8979 tablet_bootstrap.cc:492] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:19.315673 8982 ts_tablet_manager.cc:1367] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240502 01:57:19.315704 8979 ts_tablet_manager.cc:1367] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.017s
I20240502 01:57:19.316525 8979 raft_consensus.cc:348] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:19.316589 8982 raft_consensus.cc:348] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:19.316759 8982 raft_consensus.cc:374] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:19.316766 8979 raft_consensus.cc:374] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:19.316857 8982 raft_consensus.cc:729] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:19.316864 8979 raft_consensus.cc:729] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:19.317117 8984 tablet_bootstrap.cc:492] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:19.317127 8982 consensus_queue.cc:260] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:19.317287 8984 ts_tablet_manager.cc:1367] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 01:57:19.317160 8979 consensus_queue.cc:260] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:19.318143 8984 raft_consensus.cc:348] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:19.318382 8984 raft_consensus.cc:374] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:19.318482 8984 raft_consensus.cc:729] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:19.318753 8979 ts_tablet_manager.cc:1398] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 01:57:19.318733 8984 consensus_queue.cc:260] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:19.319087 8979 tablet_bootstrap.cc:492] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:19.319113 8982 ts_tablet_manager.cc:1398] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:57:19.319820 8982 tablet_bootstrap.cc:492] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:19.320825 8979 tablet_bootstrap.cc:654] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:19.321534 8982 tablet_bootstrap.cc:654] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:19.325471 8984 ts_tablet_manager.cc:1398] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:57:19.328977 8984 tablet_bootstrap.cc:492] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:19.329233 8980 tablet_bootstrap.cc:492] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:19.329421 8980 ts_tablet_manager.cc:1367] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.006s
I20240502 01:57:19.330384 8980 raft_consensus.cc:348] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:19.330651 8980 raft_consensus.cc:374] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:19.330768 8980 raft_consensus.cc:729] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:19.331074 8980 consensus_queue.cc:260] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:19.331766 8980 ts_tablet_manager.cc:1398] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:57:19.332070 8980 tablet_bootstrap.cc:492] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:19.333576 8980 tablet_bootstrap.cc:654] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:19.334110 8982 tablet_bootstrap.cc:492] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:19.334362 8982 ts_tablet_manager.cc:1367] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240502 01:57:19.334434 8984 tablet_bootstrap.cc:654] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:19.335662 8982 raft_consensus.cc:348] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:19.336108 8982 raft_consensus.cc:374] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:19.336247 8982 raft_consensus.cc:729] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:19.336815 8982 consensus_queue.cc:260] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:19.340689 8984 tablet_bootstrap.cc:492] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:19.340783 8979 tablet_bootstrap.cc:492] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:19.341046 8979 ts_tablet_manager.cc:1367] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.009s
I20240502 01:57:19.341049 8984 ts_tablet_manager.cc:1367] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240502 01:57:19.341943 8982 ts_tablet_manager.cc:1398] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:57:19.342139 8979 raft_consensus.cc:348] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:19.342465 8979 raft_consensus.cc:374] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:19.342311 8984 raft_consensus.cc:348] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:19.342657 8984 raft_consensus.cc:374] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:19.342582 8979 raft_consensus.cc:729] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:19.342765 8984 raft_consensus.cc:729] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:19.343114 8979 consensus_queue.cc:260] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:19.343132 8984 consensus_queue.cc:260] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:19.343807 8984 ts_tablet_manager.cc:1398] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:57:19.344802 8979 ts_tablet_manager.cc:1398] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:57:19.345149 8979 tablet_bootstrap.cc:492] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:19.347303 8980 tablet_bootstrap.cc:492] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:19.347520 8980 ts_tablet_manager.cc:1367] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20240502 01:57:19.348372 8980 raft_consensus.cc:348] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:19.348691 8980 raft_consensus.cc:374] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:19.348729 8979 tablet_bootstrap.cc:654] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:19.348793 8980 raft_consensus.cc:729] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:19.349095 8980 consensus_queue.cc:260] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:19.349741 8980 ts_tablet_manager.cc:1398] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:19.352202 8979 tablet_bootstrap.cc:492] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:19.352378 8979 ts_tablet_manager.cc:1367] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 01:57:19.353166 8979 raft_consensus.cc:348] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:19.353332 8979 raft_consensus.cc:374] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:19.353420 8979 raft_consensus.cc:729] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:19.353688 8979 consensus_queue.cc:260] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:19.354223 8979 ts_tablet_manager.cc:1398] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:57:19.367105 8755 raft_consensus.cc:482] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:19.367374 8755 raft_consensus.cc:504] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:19.368077 8755 leader_election.cc:288] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:19.368781 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4112f382c8e24680ae2350d9d5d8af54" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786" is_pre_election: true
I20240502 01:57:19.368811 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4112f382c8e24680ae2350d9d5d8af54" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d" is_pre_election: true
I20240502 01:57:19.369122 7646 raft_consensus.cc:2454] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 0.
I20240502 01:57:19.369102 7515 raft_consensus.cc:2454] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 0.
I20240502 01:57:19.369711 7313 leader_election.cc:302] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:19.370054 8755 raft_consensus.cc:2781] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:19.370167 8755 raft_consensus.cc:482] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:19.370285 8755 raft_consensus.cc:3037] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:19.372362 8755 raft_consensus.cc:504] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:19.373062 8755 leader_election.cc:288] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:19.373616 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4112f382c8e24680ae2350d9d5d8af54" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:19.373641 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4112f382c8e24680ae2350d9d5d8af54" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:19.373855 7515 raft_consensus.cc:3037] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:19.373883 7646 raft_consensus.cc:3037] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:19.376868 7646 raft_consensus.cc:2454] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 1.
I20240502 01:57:19.376874 7515 raft_consensus.cc:2454] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 1.
I20240502 01:57:19.377399 7313 leader_election.cc:302] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters:
I20240502 01:57:19.377739 8755 raft_consensus.cc:2781] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:19.377918 8755 raft_consensus.cc:686] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:19.378192 8755 consensus_queue.cc:237] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:19.381584 7229 catalog_manager.cc:5484] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:19.417604 8593 raft_consensus.cc:482] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:19.417837 8593 raft_consensus.cc:504] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:19.418532 8593 leader_election.cc:288] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:19.419131 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c93a84c310742b999bbb053d687639f" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412" is_pre_election: true
I20240502 01:57:19.419119 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c93a84c310742b999bbb053d687639f" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5" is_pre_election: true
I20240502 01:57:19.419399 7780 raft_consensus.cc:2454] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:19.419432 7379 raft_consensus.cc:2454] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:19.419926 7579 leader_election.cc:302] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:19.420254 8593 raft_consensus.cc:2781] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:19.420360 8593 raft_consensus.cc:482] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:19.420480 8593 raft_consensus.cc:3037] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:19.422698 8593 raft_consensus.cc:504] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:19.423333 8593 leader_election.cc:288] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:19.423873 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c93a84c310742b999bbb053d687639f" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:19.423873 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c93a84c310742b999bbb053d687639f" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:19.424129 7780 raft_consensus.cc:3037] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:19.424129 7379 raft_consensus.cc:3037] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:19.427186 7379 raft_consensus.cc:2454] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:19.427173 7780 raft_consensus.cc:2454] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:19.427742 7579 leader_election.cc:302] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:19.428072 8593 raft_consensus.cc:2781] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:19.428278 8593 raft_consensus.cc:686] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:19.428573 8593 consensus_queue.cc:237] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:19.431938 7229 catalog_manager.cc:5484] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 1 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:19.487273 8593 raft_consensus.cc:482] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:19.487541 8593 raft_consensus.cc:504] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:19.488258 8593 leader_election.cc:288] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:19.489019 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15e07df96bd94ef8b505a7af2ac88db4" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d" is_pre_election: true
I20240502 01:57:19.489063 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15e07df96bd94ef8b505a7af2ac88db4" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412" is_pre_election: true
I20240502 01:57:19.489322 7515 raft_consensus.cc:2454] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:19.489334 7780 raft_consensus.cc:2454] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:19.489868 7579 leader_election.cc:302] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:19.490176 8593 raft_consensus.cc:2781] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:19.490343 8593 raft_consensus.cc:482] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:19.490448 8593 raft_consensus.cc:3037] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:19.492645 8593 raft_consensus.cc:504] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:19.493311 8593 leader_election.cc:288] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:19.493813 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15e07df96bd94ef8b505a7af2ac88db4" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:19.493801 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15e07df96bd94ef8b505a7af2ac88db4" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:19.494032 7780 raft_consensus.cc:3037] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:19.494057 7515 raft_consensus.cc:3037] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:19.497014 7515 raft_consensus.cc:2454] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:19.497025 7780 raft_consensus.cc:2454] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:19.497561 7581 leader_election.cc:302] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:19.497905 8593 raft_consensus.cc:2781] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:19.498044 8593 raft_consensus.cc:686] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:19.498359 8593 consensus_queue.cc:237] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:19.502189 7229 catalog_manager.cc:5484] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 1 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:19.867283 8943 consensus_queue.cc:1035] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:19.874428 8943 consensus_queue.cc:1035] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:19.916818 8885 consensus_queue.cc:1035] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:19.924611 8593 consensus_queue.cc:1035] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:57:19.935529 8996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:19.936023 8996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:57:19.949465 8593 consensus_queue.cc:1035] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:57:19.950742 8996 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:57:19.960407 8593 consensus_queue.cc:1035] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
W20240502 01:57:20.200230 7581 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.212.194:39047, user_credentials={real_user=slave}} blocked reactor thread for 54628.9us
INSERT report
rows total: 2000
time total: 284.652 ms
time per row: 0.142326 ms
I20240502 01:57:20.404296 7229 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57734:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:57:20.413514 7494 tablet_service.cc:1467] Processing CreateTablet for tablet f5b90bdde42e4a9b98188f1a197ed835 (DEFAULT_TABLE table=rebalance_test_table_8 [id=1ac2c34e9bf345b096f494f1d2e6306d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:20.413719 7626 tablet_service.cc:1467] Processing CreateTablet for tablet f5b90bdde42e4a9b98188f1a197ed835 (DEFAULT_TABLE table=rebalance_test_table_8 [id=1ac2c34e9bf345b096f494f1d2e6306d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:20.414053 7494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5b90bdde42e4a9b98188f1a197ed835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:20.414225 7626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5b90bdde42e4a9b98188f1a197ed835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:20.415297 7759 tablet_service.cc:1467] Processing CreateTablet for tablet 6f65accc9dfb4bd2b8cf82cb2d3e171c (DEFAULT_TABLE table=rebalance_test_table_8 [id=1ac2c34e9bf345b096f494f1d2e6306d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:20.415762 7759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f65accc9dfb4bd2b8cf82cb2d3e171c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:20.415967 7758 tablet_service.cc:1467] Processing CreateTablet for tablet e43e7e31842d47c59425ca63bc7182da (DEFAULT_TABLE table=rebalance_test_table_8 [id=1ac2c34e9bf345b096f494f1d2e6306d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:20.416365 7758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e43e7e31842d47c59425ca63bc7182da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:20.421348 7495 tablet_service.cc:1467] Processing CreateTablet for tablet e43e7e31842d47c59425ca63bc7182da (DEFAULT_TABLE table=rebalance_test_table_8 [id=1ac2c34e9bf345b096f494f1d2e6306d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:20.421777 7495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e43e7e31842d47c59425ca63bc7182da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:20.421710 7358 tablet_service.cc:1467] Processing CreateTablet for tablet f5b90bdde42e4a9b98188f1a197ed835 (DEFAULT_TABLE table=rebalance_test_table_8 [id=1ac2c34e9bf345b096f494f1d2e6306d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:20.422178 7358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5b90bdde42e4a9b98188f1a197ed835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:20.422495 7628 tablet_service.cc:1467] Processing CreateTablet for tablet 6f65accc9dfb4bd2b8cf82cb2d3e171c (DEFAULT_TABLE table=rebalance_test_table_8 [id=1ac2c34e9bf345b096f494f1d2e6306d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:20.422937 7628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f65accc9dfb4bd2b8cf82cb2d3e171c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:20.425370 7493 tablet_service.cc:1467] Processing CreateTablet for tablet 6f65accc9dfb4bd2b8cf82cb2d3e171c (DEFAULT_TABLE table=rebalance_test_table_8 [id=1ac2c34e9bf345b096f494f1d2e6306d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:20.425745 7493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f65accc9dfb4bd2b8cf82cb2d3e171c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:20.429023 7360 tablet_service.cc:1467] Processing CreateTablet for tablet e43e7e31842d47c59425ca63bc7182da (DEFAULT_TABLE table=rebalance_test_table_8 [id=1ac2c34e9bf345b096f494f1d2e6306d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:20.429453 7360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e43e7e31842d47c59425ca63bc7182da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:20.445247 9065 tablet_bootstrap.cc:492] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:20.446771 9066 tablet_bootstrap.cc:492] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:20.447898 9067 tablet_bootstrap.cc:492] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:20.449433 9066 tablet_bootstrap.cc:654] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:20.449968 9067 tablet_bootstrap.cc:654] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:20.450114 9064 tablet_bootstrap.cc:492] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:20.453874 9064 tablet_bootstrap.cc:654] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:20.457194 9066 tablet_bootstrap.cc:492] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:20.457499 9066 ts_tablet_manager.cc:1367] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20240502 01:57:20.457967 9065 tablet_bootstrap.cc:654] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:20.458530 9066 raft_consensus.cc:348] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:20.458920 9066 raft_consensus.cc:374] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:20.459096 9066 raft_consensus.cc:729] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:20.459595 9066 consensus_queue.cc:260] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:20.460608 9066 ts_tablet_manager.cc:1398] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:57:20.464380 9066 tablet_bootstrap.cc:492] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:20.470682 9065 tablet_bootstrap.cc:492] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:20.470938 9065 ts_tablet_manager.cc:1367] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.000s
I20240502 01:57:20.471853 9065 raft_consensus.cc:348] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:20.472100 9065 raft_consensus.cc:374] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:20.472010 9064 tablet_bootstrap.cc:492] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:20.472201 9065 raft_consensus.cc:729] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:20.472462 9064 ts_tablet_manager.cc:1367] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20240502 01:57:20.472530 9065 consensus_queue.cc:260] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:20.473783 9064 raft_consensus.cc:348] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:20.474102 9064 raft_consensus.cc:374] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:20.474292 9064 raft_consensus.cc:729] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:20.474675 9064 consensus_queue.cc:260] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:20.475787 9064 ts_tablet_manager.cc:1398] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:57:20.476198 9064 tablet_bootstrap.cc:492] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:20.476523 9066 tablet_bootstrap.cc:654] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:20.476697 9065 ts_tablet_manager.cc:1398] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:57:20.479660 9064 tablet_bootstrap.cc:654] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:20.479801 9065 tablet_bootstrap.cc:492] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:20.481575 9065 tablet_bootstrap.cc:654] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:20.490249 9066 tablet_bootstrap.cc:492] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:20.490321 9064 tablet_bootstrap.cc:492] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:20.490530 9064 ts_tablet_manager.cc:1367] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20240502 01:57:20.490587 9066 ts_tablet_manager.cc:1367] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.000s
I20240502 01:57:20.490937 9065 tablet_bootstrap.cc:492] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:20.491132 9065 ts_tablet_manager.cc:1367] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240502 01:57:20.491477 9064 raft_consensus.cc:348] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.491739 9064 raft_consensus.cc:374] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:20.491832 9064 raft_consensus.cc:729] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:20.491652 9066 raft_consensus.cc:348] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.491969 9065 raft_consensus.cc:348] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.492228 9065 raft_consensus.cc:374] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:20.492132 9064 consensus_queue.cc:260] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.492228 9066 raft_consensus.cc:374] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:20.492312 9065 raft_consensus.cc:729] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:20.492439 9066 raft_consensus.cc:729] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:20.492609 9065 consensus_queue.cc:260] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.492772 9064 ts_tablet_manager.cc:1398] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:57:20.493949 9067 tablet_bootstrap.cc:492] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:20.494127 9067 ts_tablet_manager.cc:1367] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.047s user 0.013s sys 0.013s
I20240502 01:57:20.494728 9064 tablet_bootstrap.cc:492] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:20.495035 9067 raft_consensus.cc:348] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.495255 9067 raft_consensus.cc:374] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:20.495365 9067 raft_consensus.cc:729] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:20.495608 9067 consensus_queue.cc:260] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.496234 9067 ts_tablet_manager.cc:1398] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:57:20.496253 9066 consensus_queue.cc:260] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.496574 9067 tablet_bootstrap.cc:492] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:20.497941 9066 ts_tablet_manager.cc:1398] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:57:20.498363 9067 tablet_bootstrap.cc:654] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:20.501670 9064 tablet_bootstrap.cc:654] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:20.507586 9065 ts_tablet_manager.cc:1398] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240502 01:57:20.509719 9064 tablet_bootstrap.cc:492] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:20.509979 9064 ts_tablet_manager.cc:1367] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.004s
I20240502 01:57:20.510624 9067 tablet_bootstrap.cc:492] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:20.510815 9067 ts_tablet_manager.cc:1367] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20240502 01:57:20.511057 9064 raft_consensus.cc:348] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.511303 9064 raft_consensus.cc:374] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:20.511458 9064 raft_consensus.cc:729] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:20.511533 9067 raft_consensus.cc:348] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.511767 9067 raft_consensus.cc:374] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:20.511746 9064 consensus_queue.cc:260] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.511857 9067 raft_consensus.cc:729] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:20.512158 9067 consensus_queue.cc:260] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.512343 9064 ts_tablet_manager.cc:1398] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:57:20.512832 9067 ts_tablet_manager.cc:1398] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:57:20.563968 8710 raft_consensus.cc:482] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:20.564167 8710 raft_consensus.cc:504] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:20.564826 8710 leader_election.cc:288] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:20.565392 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b90bdde42e4a9b98188f1a197ed835" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786" is_pre_election: true
I20240502 01:57:20.565426 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b90bdde42e4a9b98188f1a197ed835" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5" is_pre_election: true
I20240502 01:57:20.565613 7646 raft_consensus.cc:2454] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 0.
I20240502 01:57:20.565656 7379 raft_consensus.cc:2454] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 0.
I20240502 01:57:20.566123 7451 leader_election.cc:302] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:20.566434 8710 raft_consensus.cc:2781] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:20.566567 8710 raft_consensus.cc:482] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:20.566668 8710 raft_consensus.cc:3037] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:20.567791 9060 raft_consensus.cc:482] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:20.567950 9060 raft_consensus.cc:504] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.568603 9060 leader_election.cc:288] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:20.569092 8710 raft_consensus.cc:504] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:20.569278 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43e7e31842d47c59425ca63bc7182da" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d" is_pre_election: true
I20240502 01:57:20.569309 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43e7e31842d47c59425ca63bc7182da" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5" is_pre_election: true
I20240502 01:57:20.569531 7379 raft_consensus.cc:2454] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 0.
I20240502 01:57:20.569541 7516 raft_consensus.cc:2454] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 0.
I20240502 01:57:20.569869 8710 leader_election.cc:288] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:20.570159 7713 leader_election.cc:302] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:20.570287 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b90bdde42e4a9b98188f1a197ed835" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:20.570441 9060 raft_consensus.cc:2781] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:20.570501 7646 raft_consensus.cc:3037] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:20.570576 9060 raft_consensus.cc:482] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:20.570678 9060 raft_consensus.cc:3037] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:20.571280 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b90bdde42e4a9b98188f1a197ed835" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:20.571503 7379 raft_consensus.cc:3037] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:20.572811 7646 raft_consensus.cc:2454] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 1.
I20240502 01:57:20.573289 7451 leader_election.cc:302] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:20.573683 8710 raft_consensus.cc:2781] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:20.573580 9060 raft_consensus.cc:504] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.574200 8710 raft_consensus.cc:686] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 1 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:20.574290 9060 leader_election.cc:288] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:20.574333 7379 raft_consensus.cc:2454] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 1.
I20240502 01:57:20.575124 8710 consensus_queue.cc:237] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:20.575412 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43e7e31842d47c59425ca63bc7182da" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:20.575620 7379 raft_consensus.cc:3037] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:20.575811 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43e7e31842d47c59425ca63bc7182da" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:20.576059 7515 raft_consensus.cc:3037] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:20.578876 7379 raft_consensus.cc:2454] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 1.
I20240502 01:57:20.578814 7229 catalog_manager.cc:5484] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 0 to 1, leader changed from <none> to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 1 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:20.579138 7515 raft_consensus.cc:2454] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 1.
I20240502 01:57:20.579540 7713 leader_election.cc:302] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:20.579973 9060 raft_consensus.cc:2781] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:20.580987 9060 raft_consensus.cc:686] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [term 1 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:20.581264 9060 consensus_queue.cc:237] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.585075 7229 catalog_manager.cc:5484] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 1 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } }
I20240502 01:57:20.603227 8885 raft_consensus.cc:482] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:20.603426 8885 raft_consensus.cc:504] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.604269 8885 leader_election.cc:288] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:20.604562 9060 raft_consensus.cc:482] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:20.604813 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f65accc9dfb4bd2b8cf82cb2d3e171c" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412" is_pre_election: true
I20240502 01:57:20.604810 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f65accc9dfb4bd2b8cf82cb2d3e171c" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d" is_pre_election: true
I20240502 01:57:20.604790 9060 raft_consensus.cc:504] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.605129 7515 raft_consensus.cc:2454] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:20.605326 7780 raft_consensus.cc:2454] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 0.
I20240502 01:57:20.605625 7581 leader_election.cc:302] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:20.605692 9060 leader_election.cc:288] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:20.605968 8885 raft_consensus.cc:2781] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:20.606164 8885 raft_consensus.cc:482] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:20.606328 8885 raft_consensus.cc:3037] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:20.606267 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f65accc9dfb4bd2b8cf82cb2d3e171c" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786" is_pre_election: true
I20240502 01:57:20.606922 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f65accc9dfb4bd2b8cf82cb2d3e171c" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d" is_pre_election: true
I20240502 01:57:20.607131 7515 raft_consensus.cc:2454] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 0.
I20240502 01:57:20.607553 7713 leader_election.cc:302] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:20.607877 9060 raft_consensus.cc:2781] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:20.607957 9060 raft_consensus.cc:482] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:20.608067 9060 raft_consensus.cc:3037] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:20.609066 8885 raft_consensus.cc:504] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.609580 7646 raft_consensus.cc:2379] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fb68cc03fe5644e0b4846a80d385c412 in current term 1: Already voted for candidate e8314214a0484251964e4b83a87b6786 in this term.
I20240502 01:57:20.609755 8885 leader_election.cc:288] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:20.610302 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f65accc9dfb4bd2b8cf82cb2d3e171c" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:20.610509 9060 raft_consensus.cc:504] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.610931 7780 raft_consensus.cc:2379] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e8314214a0484251964e4b83a87b6786 in current term 1: Already voted for candidate fb68cc03fe5644e0b4846a80d385c412 in this term.
I20240502 01:57:20.611033 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f65accc9dfb4bd2b8cf82cb2d3e171c" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:20.611304 7515 raft_consensus.cc:3037] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:20.611531 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f65accc9dfb4bd2b8cf82cb2d3e171c" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:20.611642 9060 leader_election.cc:288] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:20.611814 7646 raft_consensus.cc:2379] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fb68cc03fe5644e0b4846a80d385c412 in current term 1: Already voted for candidate e8314214a0484251964e4b83a87b6786 in this term.
I20240502 01:57:20.612192 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f65accc9dfb4bd2b8cf82cb2d3e171c" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:20.612721 7713 leader_election.cc:302] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb68cc03fe5644e0b4846a80d385c412; no voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786
I20240502 01:57:20.613032 9060 raft_consensus.cc:2726] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:57:20.613826 7515 raft_consensus.cc:2454] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 1.
I20240502 01:57:20.614202 7581 leader_election.cc:302] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters: fb68cc03fe5644e0b4846a80d385c412
I20240502 01:57:20.614524 8885 raft_consensus.cc:2781] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:20.614691 8885 raft_consensus.cc:686] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:20.614977 8885 consensus_queue.cc:237] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:20.618837 7229 catalog_manager.cc:5484] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 1 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } }
I20240502 01:57:20.973773 8710 consensus_queue.cc:1035] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:20.980651 8710 consensus_queue.cc:1035] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:57:21.014437 9083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:21.014840 9083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:21.022966 9083 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:57:21.033111 9060 consensus_queue.cc:1035] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:21.036707 8885 consensus_queue.cc:1035] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:21.042328 9047 consensus_queue.cc:1035] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:21.047015 8958 consensus_queue.cc:1035] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
INSERT report
rows total: 2000
time total: 267.327 ms
time per row: 0.133664 ms
I20240502 01:57:21.454777 7228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57734:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:57:21.471459 7358 tablet_service.cc:1467] Processing CreateTablet for tablet b443becac180479abb1071e4c829fbcb (DEFAULT_TABLE table=rebalance_test_table_9 [id=585594c8c5cb437295c4284577251435]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:21.472038 7358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b443becac180479abb1071e4c829fbcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:21.472446 7628 tablet_service.cc:1467] Processing CreateTablet for tablet c94f99eccbe34c9ebf3a375e9230994d (DEFAULT_TABLE table=rebalance_test_table_9 [id=585594c8c5cb437295c4284577251435]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:21.472612 7758 tablet_service.cc:1467] Processing CreateTablet for tablet b443becac180479abb1071e4c829fbcb (DEFAULT_TABLE table=rebalance_test_table_9 [id=585594c8c5cb437295c4284577251435]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:21.472771 7759 tablet_service.cc:1467] Processing CreateTablet for tablet c94f99eccbe34c9ebf3a375e9230994d (DEFAULT_TABLE table=rebalance_test_table_9 [id=585594c8c5cb437295c4284577251435]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:21.472945 7628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94f99eccbe34c9ebf3a375e9230994d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:21.473205 7759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94f99eccbe34c9ebf3a375e9230994d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:21.473752 7758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b443becac180479abb1071e4c829fbcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:21.475515 7626 tablet_service.cc:1467] Processing CreateTablet for tablet b443becac180479abb1071e4c829fbcb (DEFAULT_TABLE table=rebalance_test_table_9 [id=585594c8c5cb437295c4284577251435]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:21.475978 7626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b443becac180479abb1071e4c829fbcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:21.477838 7627 tablet_service.cc:1467] Processing CreateTablet for tablet f8f40cb87e054bb28ea0ab7aa9db20ff (DEFAULT_TABLE table=rebalance_test_table_9 [id=585594c8c5cb437295c4284577251435]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:21.478365 7627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f40cb87e054bb28ea0ab7aa9db20ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:21.479632 7360 tablet_service.cc:1467] Processing CreateTablet for tablet c94f99eccbe34c9ebf3a375e9230994d (DEFAULT_TABLE table=rebalance_test_table_9 [id=585594c8c5cb437295c4284577251435]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:21.480093 7360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94f99eccbe34c9ebf3a375e9230994d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:21.484601 7494 tablet_service.cc:1467] Processing CreateTablet for tablet f8f40cb87e054bb28ea0ab7aa9db20ff (DEFAULT_TABLE table=rebalance_test_table_9 [id=585594c8c5cb437295c4284577251435]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:21.485158 7494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f40cb87e054bb28ea0ab7aa9db20ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:21.495800 7361 tablet_service.cc:1467] Processing CreateTablet for tablet f8f40cb87e054bb28ea0ab7aa9db20ff (DEFAULT_TABLE table=rebalance_test_table_9 [id=585594c8c5cb437295c4284577251435]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:21.496306 7361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f40cb87e054bb28ea0ab7aa9db20ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:21.504073 9155 tablet_bootstrap.cc:492] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:21.509496 9155 tablet_bootstrap.cc:654] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:21.511838 9156 tablet_bootstrap.cc:492] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d: Bootstrap starting.
I20240502 01:57:21.512073 9153 tablet_bootstrap.cc:492] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:21.514142 9153 tablet_bootstrap.cc:654] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:21.515601 9156 tablet_bootstrap.cc:654] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:21.515812 9152 tablet_bootstrap.cc:492] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:21.517892 9152 tablet_bootstrap.cc:654] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:21.518891 9155 tablet_bootstrap.cc:492] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:21.519094 9155 ts_tablet_manager.cc:1367] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240502 01:57:21.519942 9153 tablet_bootstrap.cc:492] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:21.520133 9153 ts_tablet_manager.cc:1367] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 01:57:21.520087 9155 raft_consensus.cc:348] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:21.520345 9155 raft_consensus.cc:374] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:21.520452 9155 raft_consensus.cc:729] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:21.520738 9155 consensus_queue.cc:260] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:21.521093 9153 raft_consensus.cc:348] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:21.521376 9153 raft_consensus.cc:374] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:21.521526 9153 raft_consensus.cc:729] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:21.521777 9153 consensus_queue.cc:260] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:21.521944 9152 tablet_bootstrap.cc:492] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:21.522450 9153 ts_tablet_manager.cc:1398] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:21.522897 9153 tablet_bootstrap.cc:492] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:21.522115 9152 ts_tablet_manager.cc:1367] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:57:21.524175 9152 raft_consensus.cc:348] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:21.524420 9152 raft_consensus.cc:374] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:21.524526 9152 raft_consensus.cc:729] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:21.524637 9153 tablet_bootstrap.cc:654] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:21.524789 9152 consensus_queue.cc:260] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:21.526618 9152 ts_tablet_manager.cc:1398] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:57:21.526945 9152 tablet_bootstrap.cc:492] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412: Bootstrap starting.
I20240502 01:57:21.527207 9155 ts_tablet_manager.cc:1398] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:57:21.527447 9156 tablet_bootstrap.cc:492] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d: No bootstrap required, opened a new log
I20240502 01:57:21.527679 9156 ts_tablet_manager.cc:1367] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.004s
I20240502 01:57:21.528756 9152 tablet_bootstrap.cc:654] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:21.530273 9155 tablet_bootstrap.cc:492] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:21.528815 9156 raft_consensus.cc:348] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:21.533473 9153 tablet_bootstrap.cc:492] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:21.533969 9153 ts_tablet_manager.cc:1367] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 01:57:21.534431 9156 raft_consensus.cc:374] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:21.534655 9156 raft_consensus.cc:729] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Initialized, Role: FOLLOWER
I20240502 01:57:21.535010 9153 raft_consensus.cc:348] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:21.535256 9153 raft_consensus.cc:374] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:21.535343 9153 raft_consensus.cc:729] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:21.535599 9153 consensus_queue.cc:260] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:21.535912 9156 consensus_queue.cc:260] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:21.536442 9153 ts_tablet_manager.cc:1398] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:57:21.536823 9153 tablet_bootstrap.cc:492] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786: Bootstrap starting.
I20240502 01:57:21.537086 9155 tablet_bootstrap.cc:654] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:21.537406 9156 ts_tablet_manager.cc:1398] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240502 01:57:21.538784 9153 tablet_bootstrap.cc:654] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:21.540656 9155 tablet_bootstrap.cc:492] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:21.541021 9155 ts_tablet_manager.cc:1367] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20240502 01:57:21.542186 9155 raft_consensus.cc:348] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:21.542531 9155 raft_consensus.cc:374] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:21.542694 9155 raft_consensus.cc:729] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:21.543140 9155 consensus_queue.cc:260] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:21.546804 9155 ts_tablet_manager.cc:1398] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:57:21.547240 9155 tablet_bootstrap.cc:492] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5: Bootstrap starting.
I20240502 01:57:21.551172 9152 tablet_bootstrap.cc:492] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412: No bootstrap required, opened a new log
I20240502 01:57:21.551360 9153 tablet_bootstrap.cc:492] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786: No bootstrap required, opened a new log
I20240502 01:57:21.551406 9152 ts_tablet_manager.cc:1367] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.000s
I20240502 01:57:21.551525 9153 ts_tablet_manager.cc:1367] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20240502 01:57:21.552270 9152 raft_consensus.cc:348] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:21.552397 9153 raft_consensus.cc:348] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:21.552553 9152 raft_consensus.cc:374] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:21.552624 9153 raft_consensus.cc:374] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:21.552642 9152 raft_consensus.cc:729] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Initialized, Role: FOLLOWER
I20240502 01:57:21.552728 9153 raft_consensus.cc:729] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Initialized, Role: FOLLOWER
I20240502 01:57:21.552917 9152 consensus_queue.cc:260] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:21.552996 9153 consensus_queue.cc:260] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:21.553781 9153 ts_tablet_manager.cc:1398] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:57:21.553938 9155 tablet_bootstrap.cc:654] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:21.554355 9152 ts_tablet_manager.cc:1398] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:57:21.556753 9155 tablet_bootstrap.cc:492] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5: No bootstrap required, opened a new log
I20240502 01:57:21.556946 9155 ts_tablet_manager.cc:1367] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20240502 01:57:21.557804 9155 raft_consensus.cc:348] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:21.558032 9155 raft_consensus.cc:374] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:21.558429 9155 raft_consensus.cc:729] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Initialized, Role: FOLLOWER
I20240502 01:57:21.558902 9155 consensus_queue.cc:260] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:21.561280 9155 ts_tablet_manager.cc:1398] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:57:21.592372 8755 raft_consensus.cc:482] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:21.592557 8755 raft_consensus.cc:504] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:21.593258 8755 leader_election.cc:288] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:21.593837 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94f99eccbe34c9ebf3a375e9230994d" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786" is_pre_election: true
I20240502 01:57:21.593886 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94f99eccbe34c9ebf3a375e9230994d" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412" is_pre_election: true
I20240502 01:57:21.594146 7646 raft_consensus.cc:2454] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 0.
I20240502 01:57:21.594143 7779 raft_consensus.cc:2454] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 0.
I20240502 01:57:21.594637 7311 leader_election.cc:302] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:21.594987 8755 raft_consensus.cc:2781] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:21.595127 8755 raft_consensus.cc:482] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:21.595234 8755 raft_consensus.cc:3037] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:21.597272 8755 raft_consensus.cc:504] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:21.597896 8755 leader_election.cc:288] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:21.598441 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94f99eccbe34c9ebf3a375e9230994d" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:21.598466 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94f99eccbe34c9ebf3a375e9230994d" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:21.598673 7646 raft_consensus.cc:3037] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:21.598690 7779 raft_consensus.cc:3037] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:21.600647 7646 raft_consensus.cc:2454] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 1.
I20240502 01:57:21.600647 7779 raft_consensus.cc:2454] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 1.
I20240502 01:57:21.601164 7311 leader_election.cc:302] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:21.601496 8755 raft_consensus.cc:2781] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:21.601892 8755 raft_consensus.cc:686] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:21.602239 8755 consensus_queue.cc:237] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:21.605937 7233 catalog_manager.cc:5484] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 1 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:21.640429 8958 raft_consensus.cc:482] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:21.640710 8958 raft_consensus.cc:504] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:21.641491 8958 leader_election.cc:288] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:21.642138 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b443becac180479abb1071e4c829fbcb" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786" is_pre_election: true
I20240502 01:57:21.642174 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b443becac180479abb1071e4c829fbcb" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5" is_pre_election: true
I20240502 01:57:21.642388 7646 raft_consensus.cc:2454] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 0.
I20240502 01:57:21.642403 7381 raft_consensus.cc:2454] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 0.
I20240502 01:57:21.642897 7714 leader_election.cc:302] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:21.643249 8958 raft_consensus.cc:2781] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:21.643357 8958 raft_consensus.cc:482] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:21.643445 8958 raft_consensus.cc:3037] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:21.645386 8958 raft_consensus.cc:504] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:21.646039 8958 leader_election.cc:288] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:21.646589 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b443becac180479abb1071e4c829fbcb" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:21.646624 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b443becac180479abb1071e4c829fbcb" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:21.646801 7646 raft_consensus.cc:3037] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:21.646847 7381 raft_consensus.cc:3037] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:21.649039 7646 raft_consensus.cc:2454] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 1.
I20240502 01:57:21.649210 7381 raft_consensus.cc:2454] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 1.
I20240502 01:57:21.649724 7713 leader_election.cc:302] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:21.650084 8958 raft_consensus.cc:2781] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:21.650272 8958 raft_consensus.cc:686] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 1 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:21.650552 8958 consensus_queue.cc:237] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:21.654335 7233 catalog_manager.cc:5484] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 1 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:21.911387 9079 raft_consensus.cc:482] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:21.911641 9079 raft_consensus.cc:504] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:21.912458 9079 leader_election.cc:288] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:21.913087 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f40cb87e054bb28ea0ab7aa9db20ff" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786" is_pre_election: true
I20240502 01:57:21.913105 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f40cb87e054bb28ea0ab7aa9db20ff" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5" is_pre_election: true
I20240502 01:57:21.913383 7646 raft_consensus.cc:2454] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 0.
I20240502 01:57:21.913408 7381 raft_consensus.cc:2454] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 0.
I20240502 01:57:21.913924 7450 leader_election.cc:302] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:21.914274 9079 raft_consensus.cc:2781] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:21.914405 9079 raft_consensus.cc:482] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:21.914517 9079 raft_consensus.cc:3037] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:21.917364 9079 raft_consensus.cc:504] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:21.918509 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f40cb87e054bb28ea0ab7aa9db20ff" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:21.918735 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f40cb87e054bb28ea0ab7aa9db20ff" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:21.918830 7646 raft_consensus.cc:3037] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:21.919003 7381 raft_consensus.cc:3037] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:21.919215 9079 leader_election.cc:288] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:21.921805 7646 raft_consensus.cc:2454] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 1.
I20240502 01:57:21.921808 7381 raft_consensus.cc:2454] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 1.
I20240502 01:57:21.922299 7450 leader_election.cc:302] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:21.922669 9079 raft_consensus.cc:2781] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:21.923070 9079 raft_consensus.cc:686] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 1 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:21.923645 9079 consensus_queue.cc:237] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:21.927462 7233 catalog_manager.cc:5484] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 0 to 1, leader changed from <none> to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 1 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:22.072978 9060 consensus_queue.cc:1035] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:22.079702 9060 consensus_queue.cc:1035] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:22.178714 8755 consensus_queue.cc:1035] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:22.184764 8755 consensus_queue.cc:1035] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:57:22.326823 9171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:22.327181 9171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:22.335469 9171 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:57:22.350239 9079 consensus_queue.cc:1035] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:22.361456 9079 consensus_queue.cc:1035] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
INSERT report
rows total: 2000
time total: 298.235 ms
time per row: 0.149117 ms
I20240502 01:57:22.802668 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.197:43213
--local_ip_for_outbound_sockets=127.14.212.197
--tserver_master_addrs=127.14.212.254:39055
--webserver_port=42377
--webserver_interface=127.14.212.197
--builtin_ntp_servers=127.14.212.212:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:57:23.181725 9237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:23.182082 9237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:23.182204 9237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:23.191053 9237 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:57:23.191202 9237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:23.191335 9237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.197
I20240502 01:57:23.202702 9237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.197:43213
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.212.197
--webserver_port=42377
--tserver_master_addrs=127.14.212.254:39055
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.197
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:57:23.204483 9237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:23.206634 9237 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:23.221069 9242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:23.221724 9245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:23.222182 9243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:23.223023 9237 server_base.cc:1034] running on GCE node
I20240502 01:57:23.223783 9237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:23.224987 9237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:23.226284 9237 hybrid_clock.cc:648] HybridClock initialized: now 1714615043226228 us; error 47 us; skew 500 ppm
I20240502 01:57:23.226665 9237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:57:23.235291 9237 webserver.cc:460] Webserver started at http://127.14.212.197:42377/ using document root <none> and password file <none>
I20240502 01:57:23.236207 9237 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:23.236347 9237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:23.242323 9237 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 01:57:23.246280 9251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:23.248090 9237 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240502 01:57:23.248282 9237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7c2c6be28a0c47398034d89687b2d81f"
format_stamp: "Formatted at 2024-05-02 01:57:04 on dist-test-slave-mk1r"
I20240502 01:57:23.249279 9237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:23.281766 9237 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:23.282752 9237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:23.283125 9237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:23.284529 9237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:23.287513 9237 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:23.287652 9237 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:23.287817 9237 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:23.287952 9237 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240502 01:57:23.323908 9237 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.197:43213
I20240502 01:57:23.323951 9366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.197:43213 every 8 connection(s)
I20240502 01:57:23.325501 9237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:57:23.329239 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 9237
I20240502 01:57:23.329880 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.198:33529
--local_ip_for_outbound_sockets=127.14.212.198
--tserver_master_addrs=127.14.212.254:39055
--webserver_port=44493
--webserver_interface=127.14.212.198
--builtin_ntp_servers=127.14.212.212:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:57:23.343539 9367 heartbeater.cc:347] Connected to a master server at 127.14.212.254:39055
I20240502 01:57:23.343888 9367 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:23.344745 9367 heartbeater.cc:511] Master 127.14.212.254:39055 requested a full tablet report, sending...
I20240502 01:57:23.346560 7228 ts_manager.cc:194] Re-registered known tserver with Master: 7c2c6be28a0c47398034d89687b2d81f (127.14.212.197:43213)
I20240502 01:57:23.347533 7228 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.197:39925
W20240502 01:57:23.713121 9371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:23.713478 9371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:23.713634 9371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:23.722208 9371 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:57:23.722352 9371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:23.722501 9371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.198
I20240502 01:57:23.732810 9371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.198:33529
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.14.212.198
--webserver_port=44493
--tserver_master_addrs=127.14.212.254:39055
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.198
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:57:23.734532 9371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:23.736730 9371 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:23.750859 9380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:23.751392 9377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:23.751958 9378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:23.752990 9371 server_base.cc:1034] running on GCE node
I20240502 01:57:23.754204 9371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:23.755535 9371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:23.756778 9371 hybrid_clock.cc:648] HybridClock initialized: now 1714615043756717 us; error 57 us; skew 500 ppm
I20240502 01:57:23.757138 9371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:57:23.760234 9371 webserver.cc:460] Webserver started at http://127.14.212.198:44493/ using document root <none> and password file <none>
I20240502 01:57:23.761157 9371 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:23.761282 9371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:23.767303 9371 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20240502 01:57:23.771003 9386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:23.772526 9371 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20240502 01:57:23.772698 9371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "2c800a3f253f4206b0d61d9804accf57"
format_stamp: "Formatted at 2024-05-02 01:57:04 on dist-test-slave-mk1r"
I20240502 01:57:23.773629 9371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:23.807879 9371 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:23.808830 9371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:23.809173 9371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:23.810251 9371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:23.812119 9371 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:23.812203 9371 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:23.812341 9371 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:23.812417 9371 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:23.853544 9371 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.198:33529
I20240502 01:57:23.853581 9499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.198:33529 every 8 connection(s)
I20240502 01:57:23.855289 9371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:57:23.861353 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 9371
I20240502 01:57:23.862152 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.199:33829
--local_ip_for_outbound_sockets=127.14.212.199
--tserver_master_addrs=127.14.212.254:39055
--webserver_port=38583
--webserver_interface=127.14.212.199
--builtin_ntp_servers=127.14.212.212:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:57:23.871757 9500 heartbeater.cc:347] Connected to a master server at 127.14.212.254:39055
I20240502 01:57:23.872112 9500 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:23.872985 9500 heartbeater.cc:511] Master 127.14.212.254:39055 requested a full tablet report, sending...
I20240502 01:57:23.874652 7228 ts_manager.cc:194] Re-registered known tserver with Master: 2c800a3f253f4206b0d61d9804accf57 (127.14.212.198:33529)
I20240502 01:57:23.875933 7228 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.198:46987
W20240502 01:57:24.242611 9504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:24.242933 9504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:24.243039 9504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:24.251062 9504 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:57:24.251214 9504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:24.251340 9504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.199
I20240502 01:57:24.263285 9504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.199:33829
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.14.212.199
--webserver_port=38583
--tserver_master_addrs=127.14.212.254:39055
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.199
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:57:24.264628 9504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:24.266571 9504 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:24.281386 9511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:24.281873 9513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:24.282374 9510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:24.282892 9504 server_base.cc:1034] running on GCE node
I20240502 01:57:24.283641 9504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:24.284759 9504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:24.285981 9504 hybrid_clock.cc:648] HybridClock initialized: now 1714615044285928 us; error 42 us; skew 500 ppm
I20240502 01:57:24.286343 9504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:57:24.289086 9504 webserver.cc:460] Webserver started at http://127.14.212.199:38583/ using document root <none> and password file <none>
I20240502 01:57:24.289952 9504 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:24.290081 9504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:24.295953 9504 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 01:57:24.299237 9521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:24.300696 9504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:57:24.300906 9504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "7b3556854afb4b85a56c63ed84e63b02"
format_stamp: "Formatted at 2024-05-02 01:57:05 on dist-test-slave-mk1r"
I20240502 01:57:24.301810 9504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:24.320621 9504 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:24.321578 9504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:24.321931 9504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:24.323143 9504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:24.324846 9504 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:24.324970 9504 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240502 01:57:24.325110 9504 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:24.325173 9504 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:24.350332 9367 heartbeater.cc:503] Master 127.14.212.254:39055 was elected leader, sending a full tablet report...
I20240502 01:57:24.367888 9504 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.199:33829
I20240502 01:57:24.367911 9635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.199:33829 every 8 connection(s)
I20240502 01:57:24.369565 9504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:57:24.374205 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 9504
I20240502 01:57:24.374847 15187 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskITWpJv/build/asan/bin/kudu
/tmp/dist-test-taskITWpJv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.200:32919
--local_ip_for_outbound_sockets=127.14.212.200
--tserver_master_addrs=127.14.212.254:39055
--webserver_port=43075
--webserver_interface=127.14.212.200
--builtin_ntp_servers=127.14.212.212:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:57:24.386013 9636 heartbeater.cc:347] Connected to a master server at 127.14.212.254:39055
I20240502 01:57:24.386343 9636 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:24.387193 9636 heartbeater.cc:511] Master 127.14.212.254:39055 requested a full tablet report, sending...
I20240502 01:57:24.388691 7228 ts_manager.cc:194] Re-registered known tserver with Master: 7b3556854afb4b85a56c63ed84e63b02 (127.14.212.199:33829)
I20240502 01:57:24.389605 7228 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.199:38013
W20240502 01:57:24.761112 9640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:24.761442 9640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:24.761585 9640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:24.770846 9640 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:57:24.771046 9640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:24.771256 9640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.200
I20240502 01:57:24.782074 9640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.212:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.200:32919
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.14.212.200
--webserver_port=43075
--tserver_master_addrs=127.14.212.254:39055
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.200
--log_dir=/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:57:24.783479 9640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:24.785406 9640 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:24.798686 9648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:24.799237 9646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:24.799805 9645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:24.800578 9640 server_base.cc:1034] running on GCE node
I20240502 01:57:24.801311 9640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:24.802767 9640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:24.804001 9640 hybrid_clock.cc:648] HybridClock initialized: now 1714615044803926 us; error 85 us; skew 500 ppm
I20240502 01:57:24.804376 9640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:57:24.806877 9640 webserver.cc:460] Webserver started at http://127.14.212.200:43075/ using document root <none> and password file <none>
I20240502 01:57:24.807682 9640 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:24.807816 9640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:24.813944 9640 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240502 01:57:24.817142 9655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:24.818571 9640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:57:24.818748 9640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "7db0a568953d4aff999ebde5b5e820aa"
format_stamp: "Formatted at 2024-05-02 01:57:05 on dist-test-slave-mk1r"
I20240502 01:57:24.819633 9640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:24.841965 9640 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:24.843055 9640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:24.843358 9640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:24.844378 9640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:24.846325 9640 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:24.846421 9640 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:24.846556 9640 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:24.846616 9640 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:24.879105 9500 heartbeater.cc:503] Master 127.14.212.254:39055 was elected leader, sending a full tablet report...
I20240502 01:57:24.897953 9640 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.200:32919
I20240502 01:57:24.897997 9769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.212.200:32919 every 8 connection(s)
I20240502 01:57:24.899598 9640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskITWpJv/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714614748785264-15187-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:57:24.904393 15187 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskITWpJv/build/asan/bin/kudu as pid 9640
I20240502 01:57:24.920332 9770 heartbeater.cc:347] Connected to a master server at 127.14.212.254:39055
I20240502 01:57:24.920780 9770 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:24.922123 9770 heartbeater.cc:511] Master 127.14.212.254:39055 requested a full tablet report, sending...
I20240502 01:57:24.924401 7228 ts_manager.cc:194] Re-registered known tserver with Master: 7db0a568953d4aff999ebde5b5e820aa (127.14.212.200:32919)
I20240502 01:57:24.925915 7228 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.212.200:54193
I20240502 01:57:24.973861 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b443becac180479abb1071e4c829fbcb"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:24.974143 7648 raft_consensus.cc:482] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:24.974308 7648 raft_consensus.cc:3037] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:24.976437 7648 raft_consensus.cc:504] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:24.977218 7648 leader_election.cc:288] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:24.977819 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b443becac180479abb1071e4c829fbcb" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:24.977838 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b443becac180479abb1071e4c829fbcb" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:24.978076 7780 raft_consensus.cc:3032] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:24.978123 7379 raft_consensus.cc:3037] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:24.978169 7780 raft_consensus.cc:729] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:24.978578 7780 consensus_queue.cc:260] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:24.979157 7780 raft_consensus.cc:3037] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:24.980940 7379 raft_consensus.cc:2454] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:24.981714 7581 leader_election.cc:302] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:24.981772 7780 raft_consensus.cc:2454] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:24.982142 9662 raft_consensus.cc:2781] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:24.982321 9662 raft_consensus.cc:686] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:24.982693 9662 consensus_queue.cc:237] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:24.985061 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f65accc9dfb4bd2b8cf82cb2d3e171c"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:24.986228 7228 catalog_manager.cc:5484] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 1 to 2, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 2 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:24.988140 7648 raft_consensus.cc:465] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:57:24.989620 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c93a84c310742b999bbb053d687639f"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:24.989895 7648 raft_consensus.cc:465] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:57:24.990896 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "15e07df96bd94ef8b505a7af2ac88db4"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:24.991115 7648 raft_consensus.cc:465] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:57:24.992254 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4112f382c8e24680ae2350d9d5d8af54"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:24.992450 7648 raft_consensus.cc:482] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:24.992565 7648 raft_consensus.cc:3037] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:24.994563 7648 raft_consensus.cc:504] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:24.995165 7648 leader_election.cc:288] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:24.995819 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4112f382c8e24680ae2350d9d5d8af54" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:24.996083 7379 raft_consensus.cc:3032] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:24.995968 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4112f382c8e24680ae2350d9d5d8af54" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:24.996191 7379 raft_consensus.cc:729] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:24.996264 7516 raft_consensus.cc:3037] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:24.996567 7379 consensus_queue.cc:260] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:24.997133 7379 raft_consensus.cc:3037] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:24.999138 7379 raft_consensus.cc:2454] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:24.999866 7516 raft_consensus.cc:2454] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:25.000454 7581 leader_election.cc:302] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:25.000808 9662 raft_consensus.cc:2781] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.001025 9662 raft_consensus.cc:686] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.001209 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "29dce0c6f40a4358ab8855cf0c709862"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:25.001379 9662 consensus_queue.cc:237] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.001773 7648 raft_consensus.cc:482] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.001921 7648 raft_consensus.cc:3037] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.005281 7228 catalog_manager.cc:5484] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 1 to 2, leader changed from 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193) to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 2 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:25.006142 7648 raft_consensus.cc:504] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.007031 7648 leader_election.cc:288] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:25.007678 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29dce0c6f40a4358ab8855cf0c709862" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:25.007905 7516 raft_consensus.cc:3032] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.008010 7516 raft_consensus.cc:729] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.008366 7516 consensus_queue.cc:260] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.008518 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29dce0c6f40a4358ab8855cf0c709862" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.008760 7379 raft_consensus.cc:3037] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.008981 7516 raft_consensus.cc:3037] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.011029 7516 raft_consensus.cc:2454] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:25.011972 7379 raft_consensus.cc:2454] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:25.012221 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f5b90bdde42e4a9b98188f1a197ed835"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:25.012501 7648 raft_consensus.cc:482] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.012635 7648 raft_consensus.cc:3037] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.012859 7581 leader_election.cc:302] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:25.013846 9662 raft_consensus.cc:2781] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.014024 9662 raft_consensus.cc:686] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.014338 9662 consensus_queue.cc:237] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.017190 7648 raft_consensus.cc:504] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.017786 7648 leader_election.cc:288] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.018183 7228 catalog_manager.cc:5484] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 1 to 2, leader changed from d599c2145ef344fbbe26469d4331175d (127.14.212.194) to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 2 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
I20240502 01:57:25.019981 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b90bdde42e4a9b98188f1a197ed835" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:25.019989 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b90bdde42e4a9b98188f1a197ed835" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.020217 7379 raft_consensus.cc:3037] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.020341 7516 raft_consensus.cc:3032] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.020517 7516 raft_consensus.cc:729] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.021207 7516 consensus_queue.cc:260] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.021719 7516 raft_consensus.cc:3037] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.023121 7379 raft_consensus.cc:2454] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:25.023157 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "71ee1cc0eacc45149367f418f4e42954"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:25.023387 7648 raft_consensus.cc:465] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:57:25.023568 7581 leader_election.cc:302] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:25.023908 9662 raft_consensus.cc:2781] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.024117 9662 raft_consensus.cc:686] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.024747 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7ac302b18a4a41239b479f539f590b40"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:25.024672 9662 consensus_queue.cc:237] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.024971 7516 raft_consensus.cc:2454] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:25.025139 7648 raft_consensus.cc:465] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:57:25.026504 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7dad5d30968a4698abc6688c2a71cddf"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:25.026703 7648 raft_consensus.cc:482] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.026849 7648 raft_consensus.cc:3037] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.028175 7228 catalog_manager.cc:5484] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 1 to 2, leader changed from d599c2145ef344fbbe26469d4331175d (127.14.212.194) to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 2 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:25.029824 7648 raft_consensus.cc:504] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.030651 7648 leader_election.cc:288] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.031136 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dad5d30968a4698abc6688c2a71cddf" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.031402 7780 raft_consensus.cc:3032] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.031487 7780 raft_consensus.cc:729] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.031826 7780 consensus_queue.cc:260] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.032236 7780 raft_consensus.cc:3037] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.032279 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dad5d30968a4698abc6688c2a71cddf" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.032552 7379 raft_consensus.cc:3037] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.034806 7780 raft_consensus.cc:2454] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:25.035277 7579 leader_election.cc:302] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.035503 7379 raft_consensus.cc:2454] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:25.035655 9662 raft_consensus.cc:2781] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.035817 9662 raft_consensus.cc:686] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.036156 9662 consensus_queue.cc:237] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.037051 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d727fc7ec2054b46aafb44ccf4bd0ad8"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:25.037287 7648 raft_consensus.cc:465] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:57:25.039630 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c94f99eccbe34c9ebf3a375e9230994d"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:25.039896 7648 raft_consensus.cc:482] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.040091 7648 raft_consensus.cc:3037] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.040565 7228 catalog_manager.cc:5484] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 1 to 2, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 2 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:25.044183 7648 raft_consensus.cc:504] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.045159 7648 leader_election.cc:288] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:25.045666 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94f99eccbe34c9ebf3a375e9230994d" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.045907 7780 raft_consensus.cc:3037] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.045990 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94f99eccbe34c9ebf3a375e9230994d" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.046232 7379 raft_consensus.cc:3032] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.046355 7379 raft_consensus.cc:729] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:25.046684 7379 consensus_queue.cc:260] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.047199 7379 raft_consensus.cc:3037] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.048177 7780 raft_consensus.cc:2454] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:25.048666 7579 leader_election.cc:302] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.049000 9662 raft_consensus.cc:2781] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.049196 9662 raft_consensus.cc:686] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.049806 9662 consensus_queue.cc:237] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.050063 7379 raft_consensus.cc:2454] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:25.051582 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f8f40cb87e054bb28ea0ab7aa9db20ff"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:25.051934 7648 raft_consensus.cc:482] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.052196 7648 raft_consensus.cc:3037] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.056870 7648 raft_consensus.cc:504] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.058565 7648 leader_election.cc:288] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.059593 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f40cb87e054bb28ea0ab7aa9db20ff" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:25.059863 7516 raft_consensus.cc:3032] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.059710 7228 catalog_manager.cc:5484] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 1 to 2, leader changed from 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193) to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 2 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:25.059995 7516 raft_consensus.cc:729] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.060276 7516 consensus_queue.cc:260] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.060683 7516 raft_consensus.cc:3037] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.062023 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e4bb4b288a4d4fd689b750da93d2fa07"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:25.062299 7648 raft_consensus.cc:482] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.062417 7648 raft_consensus.cc:3037] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.062916 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f40cb87e054bb28ea0ab7aa9db20ff" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.063145 7379 raft_consensus.cc:3037] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.064916 7648 raft_consensus.cc:504] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.065512 7648 leader_election.cc:288] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:25.066426 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4bb4b288a4d4fd689b750da93d2fa07" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:25.066448 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4bb4b288a4d4fd689b750da93d2fa07" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.066627 7515 raft_consensus.cc:3037] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.066670 7381 raft_consensus.cc:3032] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.066767 7381 raft_consensus.cc:729] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:25.067050 7381 consensus_queue.cc:260] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.067445 7381 raft_consensus.cc:3037] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.068707 7515 raft_consensus.cc:2454] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:25.070226 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dffadd26fd38434eaf6cda23da9f1408"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:25.070452 7648 raft_consensus.cc:465] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:57:25.069932 7581 leader_election.cc:302] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:25.070477 7516 raft_consensus.cc:2454] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:25.070739 7381 raft_consensus.cc:2454] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:25.070933 9662 raft_consensus.cc:2781] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.071094 9662 raft_consensus.cc:686] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.071482 9662 consensus_queue.cc:237] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.072486 7379 raft_consensus.cc:2454] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:25.074064 7581 leader_election.cc:302] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:25.074344 9662 raft_consensus.cc:2781] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.074540 9662 raft_consensus.cc:686] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.074853 9662 consensus_queue.cc:237] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.075044 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9819568333624685aafc31d82e22f0e0"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:25.075224 7648 raft_consensus.cc:482] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.075348 7648 raft_consensus.cc:3037] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.076257 7228 catalog_manager.cc:5484] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 1 to 2, leader changed from 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193) to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 2 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
I20240502 01:57:25.080104 7648 raft_consensus.cc:504] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.082227 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9819568333624685aafc31d82e22f0e0" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.082510 7780 raft_consensus.cc:3037] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.085368 7780 raft_consensus.cc:2454] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:25.085881 7579 leader_election.cc:302] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.086213 9662 raft_consensus.cc:2781] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.086359 9662 raft_consensus.cc:686] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.086231 7228 catalog_manager.cc:5484] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 1 to 2, leader changed from d599c2145ef344fbbe26469d4331175d (127.14.212.194) to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 2 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:25.086658 9662 consensus_queue.cc:237] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.086999 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9819568333624685aafc31d82e22f0e0" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:25.087221 7516 raft_consensus.cc:3032] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.087332 7516 raft_consensus.cc:729] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.087613 7516 consensus_queue.cc:260] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.088104 7516 raft_consensus.cc:3037] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.091389 7516 raft_consensus.cc:2454] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:25.092067 7648 leader_election.cc:288] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:25.093405 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7ca112cbaf214bd7aad56b8e1a336ddd"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:25.093590 7648 raft_consensus.cc:465] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:57:25.094391 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1d6946feb1c14a1eaacad72be52108f3"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:25.094555 7648 raft_consensus.cc:465] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:57:25.094305 7228 catalog_manager.cc:5484] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 1 to 2, leader changed from d599c2145ef344fbbe26469d4331175d (127.14.212.194) to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 2 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } }
I20240502 01:57:25.095368 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3ac865cf3c044ac8ae3b0383b0b73c0"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:25.095537 7648 raft_consensus.cc:482] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.095620 7648 raft_consensus.cc:3037] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.097910 7648 raft_consensus.cc:504] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.099072 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ac865cf3c044ac8ae3b0383b0b73c0" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.099206 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ac865cf3c044ac8ae3b0383b0b73c0" candidate_uuid: "e8314214a0484251964e4b83a87b6786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.099305 7780 raft_consensus.cc:3037] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.099460 7379 raft_consensus.cc:3032] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.099565 7379 raft_consensus.cc:729] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:25.099874 7379 consensus_queue.cc:260] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.100266 7379 raft_consensus.cc:3037] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.100999 7648 leader_election.cc:288] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:25.101567 7780 raft_consensus.cc:2454] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:25.102118 7579 leader_election.cc:302] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.102440 9662 raft_consensus.cc:2781] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.102797 9662 raft_consensus.cc:686] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Leader. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.102952 7648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "04426a27d50548b9a3eb0e3eb09a5bcc"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:25.103212 7648 raft_consensus.cc:465] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:57:25.103246 9662 consensus_queue.cc:237] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.103569 7379 raft_consensus.cc:2454] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8314214a0484251964e4b83a87b6786 in term 2.
I20240502 01:57:25.104519 7646 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46549b89a28243a7b4245c14ecba6008"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:25.104795 7646 raft_consensus.cc:465] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:57:25.106348 7646 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26522a1c09bf44aca91d0410a7459b26"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:25.106532 7646 raft_consensus.cc:465] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:57:25.107554 7646 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0718691ffdc04f719ad7df57987df68c"
dest_uuid: "e8314214a0484251964e4b83a87b6786"
from {username='slave'} at 127.0.0.1:41116
I20240502 01:57:25.107497 7228 catalog_manager.cc:5484] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 reported cstate change: term changed from 1 to 2, leader changed from 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193) to e8314214a0484251964e4b83a87b6786 (127.14.212.195). New cstate: current_term: 2 leader_uuid: "e8314214a0484251964e4b83a87b6786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
I20240502 01:57:25.107733 7646 raft_consensus.cc:465] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:57:25.131431 7780 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b443becac180479abb1071e4c829fbcb"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.131870 7780 raft_consensus.cc:482] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.132040 7780 raft_consensus.cc:3037] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.135044 7780 raft_consensus.cc:504] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.135962 7780 leader_election.cc:288] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 3 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:25.136574 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b443becac180479abb1071e4c829fbcb" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.136835 7379 raft_consensus.cc:3037] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.136832 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b443becac180479abb1071e4c829fbcb" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.137111 7648 raft_consensus.cc:3032] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.137254 7648 raft_consensus.cc:729] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.137531 7780 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c94f99eccbe34c9ebf3a375e9230994d"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.137636 7648 consensus_queue.cc:260] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.137826 7780 raft_consensus.cc:482] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.138060 7780 raft_consensus.cc:3037] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.138160 7648 raft_consensus.cc:3037] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.139961 7379 raft_consensus.cc:2454] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 3.
I20240502 01:57:25.140434 7713 leader_election.cc:302] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.140789 9060 raft_consensus.cc:2781] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.141566 7648 raft_consensus.cc:2396] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fb68cc03fe5644e0b4846a80d385c412 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.141675 7780 raft_consensus.cc:504] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.142491 7780 leader_election.cc:288] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 3 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.143108 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94f99eccbe34c9ebf3a375e9230994d" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.143343 7646 raft_consensus.cc:3032] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.143461 7646 raft_consensus.cc:729] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.143553 7779 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f65accc9dfb4bd2b8cf82cb2d3e171c"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.143786 7779 raft_consensus.cc:482] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.143913 7779 raft_consensus.cc:3037] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.143784 7646 consensus_queue.cc:260] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.144131 7646 raft_consensus.cc:3037] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.146977 7646 raft_consensus.cc:2396] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fb68cc03fe5644e0b4846a80d385c412 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.146901 7779 raft_consensus.cc:504] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.147665 7779 leader_election.cc:288] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:25.147670 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94f99eccbe34c9ebf3a375e9230994d" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.147892 7379 raft_consensus.cc:3037] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.148485 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f65accc9dfb4bd2b8cf82cb2d3e171c" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:25.148758 7516 raft_consensus.cc:3037] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.149266 7647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f65accc9dfb4bd2b8cf82cb2d3e171c" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.149452 7647 raft_consensus.cc:3032] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.149539 7647 raft_consensus.cc:729] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.149819 7647 consensus_queue.cc:260] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.150223 7647 raft_consensus.cc:3037] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.151808 7516 raft_consensus.cc:2454] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.152354 7713 leader_election.cc:302] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.154390 7647 raft_consensus.cc:2454] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.155236 7779 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e43e7e31842d47c59425ca63bc7182da"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.155561 7779 raft_consensus.cc:465] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:57:25.156817 7779 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "15e07df96bd94ef8b505a7af2ac88db4"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.157212 7779 raft_consensus.cc:482] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.157441 7779 raft_consensus.cc:3037] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.156816 7379 raft_consensus.cc:2454] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 3.
I20240502 01:57:25.161419 7779 raft_consensus.cc:504] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.163177 7779 leader_election.cc:288] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:25.163628 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15e07df96bd94ef8b505a7af2ac88db4" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.165031 7780 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "71ee1cc0eacc45149367f418f4e42954"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.165046 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15e07df96bd94ef8b505a7af2ac88db4" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:25.165401 7780 raft_consensus.cc:482] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.165436 7516 raft_consensus.cc:3037] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.165658 7780 raft_consensus.cc:3037] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.168282 7648 raft_consensus.cc:3032] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.168426 7516 raft_consensus.cc:2454] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.168480 7648 raft_consensus.cc:729] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.169046 7713 leader_election.cc:302] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters: e8314214a0484251964e4b83a87b6786
I20240502 01:57:25.168953 7648 consensus_queue.cc:260] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.169852 7648 raft_consensus.cc:3037] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.170050 9796 raft_consensus.cc:2781] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.170236 9796 raft_consensus.cc:686] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.170724 9796 consensus_queue.cc:237] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.172047 7780 raft_consensus.cc:504] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.174980 7648 raft_consensus.cc:2454] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.176733 7780 leader_election.cc:288] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:25.177024 9796 raft_consensus.cc:2781] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.177201 9796 raft_consensus.cc:686] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [term 3 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.177507 9796 consensus_queue.cc:237] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.178406 7779 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7dad5d30968a4698abc6688c2a71cddf"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.178740 7779 raft_consensus.cc:482] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.178965 7779 raft_consensus.cc:3037] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.179251 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ee1cc0eacc45149367f418f4e42954" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:25.179513 7516 raft_consensus.cc:3037] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.182554 7516 raft_consensus.cc:2454] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.182988 9060 raft_consensus.cc:686] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 3 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.176370 7647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ee1cc0eacc45149367f418f4e42954" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.183362 7647 raft_consensus.cc:3032] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.183388 7713 leader_election.cc:302] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.183506 7647 raft_consensus.cc:729] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.183498 9060 consensus_queue.cc:237] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.183805 7713 leader_election.cc:302] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.182610 7779 raft_consensus.cc:504] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.183785 7647 consensus_queue.cc:260] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.184263 7647 raft_consensus.cc:3037] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.184928 9797 raft_consensus.cc:2781] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.186515 7779 leader_election.cc:288] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 3 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.187439 7647 raft_consensus.cc:2454] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.189332 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dad5d30968a4698abc6688c2a71cddf" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.189584 7646 raft_consensus.cc:3032] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.189711 7646 raft_consensus.cc:729] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.190016 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dad5d30968a4698abc6688c2a71cddf" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.189998 7646 consensus_queue.cc:260] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.190264 7379 raft_consensus.cc:3037] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.190322 7646 raft_consensus.cc:3037] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.193369 7379 raft_consensus.cc:2454] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 3.
I20240502 01:57:25.193935 7713 leader_election.cc:302] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.195104 7646 raft_consensus.cc:2396] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fb68cc03fe5644e0b4846a80d385c412 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.195902 7779 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c93a84c310742b999bbb053d687639f"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.196274 7779 raft_consensus.cc:482] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.196501 7779 raft_consensus.cc:3037] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.198164 9802 raft_consensus.cc:2781] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.198591 9795 raft_consensus.cc:2781] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.198832 9795 raft_consensus.cc:686] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 3 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.199347 9795 consensus_queue.cc:237] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.202755 7779 raft_consensus.cc:504] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.205333 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c93a84c310742b999bbb053d687639f" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.205756 7648 raft_consensus.cc:3032] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.205933 7648 raft_consensus.cc:729] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.206368 7648 consensus_queue.cc:260] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.207067 7648 raft_consensus.cc:3037] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.207581 9802 raft_consensus.cc:686] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.208068 9802 consensus_queue.cc:237] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.211079 7648 raft_consensus.cc:2454] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.211827 7714 leader_election.cc:302] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.212312 9803 raft_consensus.cc:2781] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.212551 9803 raft_consensus.cc:686] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.212769 7779 leader_election.cc:288] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.213012 9803 consensus_queue.cc:237] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.213429 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c93a84c310742b999bbb053d687639f" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.213718 7379 raft_consensus.cc:3037] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.216740 7379 raft_consensus.cc:2454] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.217420 7780 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7a14895a05814947bcec5111f41fdfb9"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.217734 7780 raft_consensus.cc:482] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.217875 7780 raft_consensus.cc:3037] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.220505 7780 raft_consensus.cc:504] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.221755 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a14895a05814947bcec5111f41fdfb9" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:25.222028 7516 raft_consensus.cc:3037] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.225085 7516 raft_consensus.cc:2454] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.225656 7713 leader_election.cc:302] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.226116 9799 raft_consensus.cc:2781] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.226317 9799 raft_consensus.cc:686] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.226761 9799 consensus_queue.cc:237] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.229156 7780 leader_election.cc:288] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.229403 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a14895a05814947bcec5111f41fdfb9" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.229805 7379 raft_consensus.cc:3032] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.230381 7779 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1b9c54c770fd47e4b92a804de3b2bcf3"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.230602 7779 raft_consensus.cc:482] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.230741 7779 raft_consensus.cc:3037] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.232371 7379 raft_consensus.cc:729] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:25.232895 7379 consensus_queue.cc:260] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.233536 7379 raft_consensus.cc:3037] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.234172 7779 raft_consensus.cc:504] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.236155 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9c54c770fd47e4b92a804de3b2bcf3" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.236457 7381 raft_consensus.cc:3037] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.236481 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9c54c770fd47e4b92a804de3b2bcf3" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:25.236760 7516 raft_consensus.cc:3032] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.236871 7516 raft_consensus.cc:729] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.237182 7516 consensus_queue.cc:260] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.237598 7516 raft_consensus.cc:3037] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.237892 7779 leader_election.cc:288] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:25.239068 7780 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c557be7764d456fa5d8a7883223951c"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.239298 7780 raft_consensus.cc:482] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.239431 7780 raft_consensus.cc:3037] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.240453 7381 raft_consensus.cc:2454] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.241828 7713 leader_election.cc:302] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.242522 9802 raft_consensus.cc:2781] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.242821 9802 raft_consensus.cc:686] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.243314 9802 consensus_queue.cc:237] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.242537 7780 raft_consensus.cc:504] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.248571 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c557be7764d456fa5d8a7883223951c" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:25.248922 7515 raft_consensus.cc:3037] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.248934 7379 raft_consensus.cc:2454] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.252565 7515 raft_consensus.cc:2454] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.253374 7713 leader_election.cc:302] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.253532 7516 raft_consensus.cc:2454] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.254788 9796 raft_consensus.cc:2781] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.256018 9796 raft_consensus.cc:686] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.256506 9796 consensus_queue.cc:237] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.254994 9797 raft_consensus.cc:686] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.257122 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c557be7764d456fa5d8a7883223951c" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.255748 7780 leader_election.cc:288] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.257807 9797 consensus_queue.cc:237] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.260176 7780 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5bcb764b10d43419e58fdcfeadb2143"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.260483 7780 raft_consensus.cc:465] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:57:25.257512 7379 raft_consensus.cc:3032] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.261754 7379 raft_consensus.cc:729] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:25.261752 7779 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7ca112cbaf214bd7aad56b8e1a336ddd"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.262022 7779 raft_consensus.cc:482] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.262162 7779 raft_consensus.cc:3037] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.262151 7379 consensus_queue.cc:260] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.262554 7379 raft_consensus.cc:3037] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.265264 7779 raft_consensus.cc:504] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.266503 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca112cbaf214bd7aad56b8e1a336ddd" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.266768 7648 raft_consensus.cc:3032] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.266911 7648 raft_consensus.cc:729] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.267203 7648 consensus_queue.cc:260] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.267354 7379 raft_consensus.cc:2454] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.267657 7648 raft_consensus.cc:3037] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.270520 7648 raft_consensus.cc:2454] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.271044 7714 leader_election.cc:302] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.271713 7779 leader_election.cc:288] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.272025 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca112cbaf214bd7aad56b8e1a336ddd" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.272298 7379 raft_consensus.cc:3037] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.272352 9797 raft_consensus.cc:2781] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.272668 9797 raft_consensus.cc:686] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.273415 9797 consensus_queue.cc:237] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.273926 7228 catalog_manager.cc:5484] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 2 to 3, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 3 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:25.274571 7228 catalog_manager.cc:5484] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 1 to 2, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 2 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:25.275058 7228 catalog_manager.cc:5484] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 1 to 2, leader changed from 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193) to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 2 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:25.275481 7379 raft_consensus.cc:2454] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.276626 7228 catalog_manager.cc:5484] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 1 to 2, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 2 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:25.277045 7779 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1d6946feb1c14a1eaacad72be52108f3"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.277279 7779 raft_consensus.cc:482] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.277429 7779 raft_consensus.cc:3037] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.277544 7228 catalog_manager.cc:5484] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 2 to 3, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 3 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:25.278134 7228 catalog_manager.cc:5484] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 1 to 2, leader changed from d599c2145ef344fbbe26469d4331175d (127.14.212.194) to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 2 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:25.278568 7228 catalog_manager.cc:5484] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 1 to 2, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 2 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } }
I20240502 01:57:25.280829 7779 raft_consensus.cc:504] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.281191 7228 catalog_manager.cc:5484] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 1 to 2, leader changed from 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193) to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 2 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:25.282122 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6946feb1c14a1eaacad72be52108f3" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.282359 7648 raft_consensus.cc:3032] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.282503 7648 raft_consensus.cc:729] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.282620 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6946feb1c14a1eaacad72be52108f3" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.282850 7379 raft_consensus.cc:3037] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.282863 7648 consensus_queue.cc:260] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.283280 7648 raft_consensus.cc:3037] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.283552 7779 leader_election.cc:288] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.286856 7379 raft_consensus.cc:2454] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.287127 7228 catalog_manager.cc:5484] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 1 to 2, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 2 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:25.287501 7713 leader_election.cc:302] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.287902 9803 raft_consensus.cc:2781] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.288093 9803 raft_consensus.cc:686] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.287814 7228 catalog_manager.cc:5484] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 2 to 3, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 3 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:25.288451 9803 consensus_queue.cc:237] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.289676 7779 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3ac865cf3c044ac8ae3b0383b0b73c0"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.289916 7779 raft_consensus.cc:482] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.290149 7648 raft_consensus.cc:2454] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.291122 7779 raft_consensus.cc:3037] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.303349 7779 raft_consensus.cc:504] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.305159 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ac865cf3c044ac8ae3b0383b0b73c0" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.305488 7379 raft_consensus.cc:3037] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.305907 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ac865cf3c044ac8ae3b0383b0b73c0" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.306135 7648 raft_consensus.cc:3032] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.306246 7648 raft_consensus.cc:729] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.306591 7648 consensus_queue.cc:260] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.306922 7648 raft_consensus.cc:3037] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.308851 7379 raft_consensus.cc:2454] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 3.
I20240502 01:57:25.309636 7713 leader_election.cc:302] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.310040 9796 raft_consensus.cc:2781] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.310127 7648 raft_consensus.cc:2396] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fb68cc03fe5644e0b4846a80d385c412 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.310243 9796 raft_consensus.cc:686] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [term 3 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.310571 9796 consensus_queue.cc:237] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.310901 7779 leader_election.cc:288] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 3 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:25.312484 7780 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dffadd26fd38434eaf6cda23da9f1408"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.312747 7780 raft_consensus.cc:482] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.312914 7780 raft_consensus.cc:3037] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.315358 7780 raft_consensus.cc:504] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.316108 7780 leader_election.cc:288] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:25.316561 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffadd26fd38434eaf6cda23da9f1408" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:25.316812 7516 raft_consensus.cc:3037] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.317304 7780 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9819568333624685aafc31d82e22f0e0"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.317502 7780 raft_consensus.cc:482] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.317639 7780 raft_consensus.cc:3037] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.320003 7780 raft_consensus.cc:504] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.320942 7516 raft_consensus.cc:2454] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.321198 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9819568333624685aafc31d82e22f0e0" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.321434 7646 raft_consensus.cc:3032] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.321455 7713 leader_election.cc:302] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.321549 7646 raft_consensus.cc:729] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.321839 9803 raft_consensus.cc:2781] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.321864 7646 consensus_queue.cc:260] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.322199 7646 raft_consensus.cc:3037] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.322216 9803 raft_consensus.cc:686] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.322613 9803 consensus_queue.cc:237] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.325526 7646 raft_consensus.cc:2396] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fb68cc03fe5644e0b4846a80d385c412 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.326118 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffadd26fd38434eaf6cda23da9f1408" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.326310 7648 raft_consensus.cc:3032] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.326392 7648 raft_consensus.cc:729] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.326671 7648 consensus_queue.cc:260] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.327040 7648 raft_consensus.cc:3037] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.329542 7228 catalog_manager.cc:5484] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 1 to 2, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 2 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:25.330036 7648 raft_consensus.cc:2454] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.330178 7228 catalog_manager.cc:5484] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 1 to 2, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 2 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:25.332396 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9819568333624685aafc31d82e22f0e0" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:25.332742 7516 raft_consensus.cc:3037] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.336309 7516 raft_consensus.cc:2454] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 3.
I20240502 01:57:25.336685 7780 leader_election.cc:288] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 3 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:25.336967 7713 leader_election.cc:302] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, fb68cc03fe5644e0b4846a80d385c412; no voters: e8314214a0484251964e4b83a87b6786
I20240502 01:57:25.337682 9796 raft_consensus.cc:2781] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.337875 9796 raft_consensus.cc:686] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [term 3 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.338150 7780 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3c56e67fc824a8799e0218b245f52c9"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.338284 9796 consensus_queue.cc:237] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.338478 7780 raft_consensus.cc:465] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:57:25.339759 7779 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3296950eda164749ab23df9d041ff456"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.339967 7779 raft_consensus.cc:482] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.340103 7779 raft_consensus.cc:3037] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.342619 7228 catalog_manager.cc:5484] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 2 to 3, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 3 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
I20240502 01:57:25.343025 7228 catalog_manager.cc:5484] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 1 to 2, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 2 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } }
I20240502 01:57:25.343376 7779 raft_consensus.cc:504] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.344235 7779 leader_election.cc:288] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.344862 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296950eda164749ab23df9d041ff456" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.345144 7379 raft_consensus.cc:3032] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.345274 7379 raft_consensus.cc:729] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:25.345577 7379 consensus_queue.cc:260] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.345813 7777 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46549b89a28243a7b4245c14ecba6008"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.345978 7379 raft_consensus.cc:3037] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.346067 7777 raft_consensus.cc:482] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.346213 7777 raft_consensus.cc:3037] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.347321 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296950eda164749ab23df9d041ff456" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:25.347537 7516 raft_consensus.cc:3037] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.349013 7379 raft_consensus.cc:2454] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.349581 7713 leader_election.cc:302] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.349905 7777 raft_consensus.cc:504] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.350996 7777 leader_election.cc:288] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.351308 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46549b89a28243a7b4245c14ecba6008" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.351569 7379 raft_consensus.cc:3037] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.351920 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46549b89a28243a7b4245c14ecba6008" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.352090 7777 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26522a1c09bf44aca91d0410a7459b26"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.352152 7646 raft_consensus.cc:3032] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.352274 7646 raft_consensus.cc:729] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.352331 7777 raft_consensus.cc:482] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.352450 7777 raft_consensus.cc:3037] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.353482 7228 catalog_manager.cc:5484] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 2 to 3, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 3 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } }
I20240502 01:57:25.354754 7646 consensus_queue.cc:260] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.355218 7646 raft_consensus.cc:3037] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.356066 7379 raft_consensus.cc:2454] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.356628 7713 leader_election.cc:302] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.356990 9803 raft_consensus.cc:2781] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.356911 7777 raft_consensus.cc:504] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.357184 9803 raft_consensus.cc:686] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.357468 9803 consensus_queue.cc:237] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.357975 9799 raft_consensus.cc:2781] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.358129 9799 raft_consensus.cc:686] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.358354 9799 consensus_queue.cc:237] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.358386 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26522a1c09bf44aca91d0410a7459b26" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:25.358628 7515 raft_consensus.cc:3037] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.361776 7515 raft_consensus.cc:2454] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.361557 7646 raft_consensus.cc:2454] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.362561 7713 leader_election.cc:302] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.362958 9802 raft_consensus.cc:2781] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.363114 9802 raft_consensus.cc:686] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.363570 9802 consensus_queue.cc:237] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.363719 7516 raft_consensus.cc:2454] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.373121 7777 leader_election.cc:288] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:25.374436 7780 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0718691ffdc04f719ad7df57987df68c"
dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
from {username='slave'} at 127.0.0.1:38520
I20240502 01:57:25.374840 7780 raft_consensus.cc:482] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.375056 7780 raft_consensus.cc:3037] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.377187 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26522a1c09bf44aca91d0410a7459b26" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.377518 7646 raft_consensus.cc:3032] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.377672 7646 raft_consensus.cc:729] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.378423 7646 consensus_queue.cc:260] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.378683 7780 raft_consensus.cc:504] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.379061 7646 raft_consensus.cc:3037] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.380093 7780 leader_election.cc:288] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:25.381335 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0718691ffdc04f719ad7df57987df68c" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.381645 7648 raft_consensus.cc:3032] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.381786 7648 raft_consensus.cc:729] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.381960 7516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0718691ffdc04f719ad7df57987df68c" candidate_uuid: "fb68cc03fe5644e0b4846a80d385c412" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:25.382366 7516 raft_consensus.cc:3037] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.383193 7648 consensus_queue.cc:260] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.384516 7648 raft_consensus.cc:3037] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.391053 7646 raft_consensus.cc:2454] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.392737 9636 heartbeater.cc:503] Master 127.14.212.254:39055 was elected leader, sending a full tablet report...
I20240502 01:57:25.395977 7228 catalog_manager.cc:5484] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 1 to 2, leader changed from 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193) to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 2 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:25.396641 7228 catalog_manager.cc:5484] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 1 to 2, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 2 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:25.398366 7516 raft_consensus.cc:2454] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.393062 7648 raft_consensus.cc:2454] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb68cc03fe5644e0b4846a80d385c412 in term 2.
I20240502 01:57:25.399396 7714 leader_election.cc:302] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.399749 9799 raft_consensus.cc:2781] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.399904 9799 raft_consensus.cc:686] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Leader. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.400213 9799 consensus_queue.cc:237] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.405988 7228 catalog_manager.cc:5484] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 1 to 2, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 2 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:25.414891 7228 catalog_manager.cc:5484] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 reported cstate change: term changed from 1 to 2, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196). New cstate: current_term: 2 leader_uuid: "fb68cc03fe5644e0b4846a80d385c412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
I20240502 01:57:25.425132 7516 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f65accc9dfb4bd2b8cf82cb2d3e171c"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.425452 7516 raft_consensus.cc:482] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.425618 7516 raft_consensus.cc:3037] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.428755 7516 raft_consensus.cc:504] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.429718 7516 leader_election.cc:288] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:25.430315 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f65accc9dfb4bd2b8cf82cb2d3e171c" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.430568 7780 raft_consensus.cc:3032] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.430667 7780 raft_consensus.cc:729] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.430603 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f65accc9dfb4bd2b8cf82cb2d3e171c" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.430881 7648 raft_consensus.cc:3037] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.430930 7780 consensus_queue.cc:260] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.431246 7780 raft_consensus.cc:3037] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.434497 7780 raft_consensus.cc:2396] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d599c2145ef344fbbe26469d4331175d for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.435853 7648 raft_consensus.cc:2454] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 3.
I20240502 01:57:25.436347 7516 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e43e7e31842d47c59425ca63bc7182da"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.436506 7451 leader_election.cc:302] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters: fb68cc03fe5644e0b4846a80d385c412
I20240502 01:57:25.436650 7516 raft_consensus.cc:482] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.436779 7516 raft_consensus.cc:3037] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.437042 9372 raft_consensus.cc:2781] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.440527 7516 raft_consensus.cc:504] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.441946 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43e7e31842d47c59425ca63bc7182da" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.442206 7780 raft_consensus.cc:3032] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.442330 7780 raft_consensus.cc:729] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.442668 7780 consensus_queue.cc:260] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.443109 7780 raft_consensus.cc:3037] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.445891 9372 raft_consensus.cc:686] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.446331 7780 raft_consensus.cc:2454] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 2.
I20240502 01:57:25.446909 7448 leader_election.cc:302] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.446794 9372 consensus_queue.cc:237] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.447467 7516 leader_election.cc:288] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 2 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.448663 7516 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4112f382c8e24680ae2350d9d5d8af54"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.448935 7516 raft_consensus.cc:482] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.449086 7516 raft_consensus.cc:3037] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.449720 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43e7e31842d47c59425ca63bc7182da" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.450042 7379 raft_consensus.cc:3037] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.452090 7516 raft_consensus.cc:504] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.453091 7516 leader_election.cc:288] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.453284 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4112f382c8e24680ae2350d9d5d8af54" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.453518 7648 raft_consensus.cc:3032] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.453600 7648 raft_consensus.cc:729] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.453878 7648 consensus_queue.cc:260] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.454161 7515 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "15e07df96bd94ef8b505a7af2ac88db4"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.454216 7648 raft_consensus.cc:3037] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.454367 7515 raft_consensus.cc:482] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.454492 7515 raft_consensus.cc:3037] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.457433 7648 raft_consensus.cc:2396] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d599c2145ef344fbbe26469d4331175d for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.457401 7515 raft_consensus.cc:504] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.457609 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4112f382c8e24680ae2350d9d5d8af54" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.457871 7381 raft_consensus.cc:3037] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.458213 7515 leader_election.cc:288] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:25.459343 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15e07df96bd94ef8b505a7af2ac88db4" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.459399 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15e07df96bd94ef8b505a7af2ac88db4" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.459535 7646 raft_consensus.cc:3037] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.459615 7780 raft_consensus.cc:3032] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.459724 7780 raft_consensus.cc:729] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.460027 7780 consensus_queue.cc:260] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.460361 7780 raft_consensus.cc:3037] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.460685 7381 raft_consensus.cc:2454] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 3.
I20240502 01:57:25.461674 7450 leader_election.cc:302] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters: e8314214a0484251964e4b83a87b6786
I20240502 01:57:25.462404 7646 raft_consensus.cc:2454] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 3.
I20240502 01:57:25.462858 7379 raft_consensus.cc:2454] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 2.
I20240502 01:57:25.463358 7780 raft_consensus.cc:2396] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d599c2145ef344fbbe26469d4331175d for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.463521 7451 leader_election.cc:302] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:25.463503 7381 raft_consensus.cc:1261] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Refusing update from remote peer e8314214a0484251964e4b83a87b6786: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:25.464257 7516 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "29dce0c6f40a4358ab8855cf0c709862"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.464603 7516 raft_consensus.cc:482] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.464759 7516 raft_consensus.cc:3037] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.465006 9662 consensus_queue.cc:1035] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.468016 7516 raft_consensus.cc:504] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.468799 7516 leader_election.cc:288] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:25.475891 9823 raft_consensus.cc:2781] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.478469 9831 raft_consensus.cc:2781] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.479351 7515 raft_consensus.cc:1261] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Refusing update from remote peer e8314214a0484251964e4b83a87b6786: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:25.479740 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29dce0c6f40a4358ab8855cf0c709862" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.480094 7382 raft_consensus.cc:3037] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.479907 9823 raft_consensus.cc:686] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.480823 9823 consensus_queue.cc:237] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.482506 7382 raft_consensus.cc:2454] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 3.
I20240502 01:57:25.483477 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29dce0c6f40a4358ab8855cf0c709862" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.483757 7648 raft_consensus.cc:3032] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.483866 7648 raft_consensus.cc:729] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.483986 7450 leader_election.cc:302] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters:
I20240502 01:57:25.484037 9831 raft_consensus.cc:686] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.484187 7648 consensus_queue.cc:260] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.484516 7648 raft_consensus.cc:3037] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.484670 7516 raft_consensus.cc:1261] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Refusing update from remote peer e8314214a0484251964e4b83a87b6786: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:25.484527 9831 consensus_queue.cc:237] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.485716 9821 raft_consensus.cc:2781] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.485911 9821 raft_consensus.cc:686] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [term 2 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.486735 9832 raft_consensus.cc:2781] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.486598 9821 consensus_queue.cc:237] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.487640 9832 raft_consensus.cc:686] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.488183 9257 consensus_queue.cc:1035] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.488025 9832 consensus_queue.cc:237] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.488627 9257 consensus_queue.cc:1035] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.491339 7648 raft_consensus.cc:2396] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d599c2145ef344fbbe26469d4331175d for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.492635 7515 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f5b90bdde42e4a9b98188f1a197ed835"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.492928 7515 raft_consensus.cc:482] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.493065 7515 raft_consensus.cc:3037] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.494452 7379 raft_consensus.cc:1261] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Refusing update from remote peer e8314214a0484251964e4b83a87b6786: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:25.495304 9257 consensus_queue.cc:1035] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.496275 7515 raft_consensus.cc:504] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.496809 7514 raft_consensus.cc:1226] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Rejecting Update request from peer e8314214a0484251964e4b83a87b6786 for earlier term 2. Current term is 3. Ops: []
I20240502 01:57:25.497536 7515 leader_election.cc:288] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.497779 9257 consensus_queue.cc:1046] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.497905 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b90bdde42e4a9b98188f1a197ed835" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.498131 9257 raft_consensus.cc:3032] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.498262 9257 raft_consensus.cc:729] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.498731 9257 consensus_queue.cc:260] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.499228 9257 raft_consensus.cc:3037] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.501710 7515 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "71ee1cc0eacc45149367f418f4e42954"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.501840 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b90bdde42e4a9b98188f1a197ed835" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.501981 7515 raft_consensus.cc:482] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.502133 7515 raft_consensus.cc:3037] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.505232 7515 raft_consensus.cc:504] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.506428 7515 leader_election.cc:288] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:25.506345 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ee1cc0eacc45149367f418f4e42954" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.506620 7646 raft_consensus.cc:3037] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.509814 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ee1cc0eacc45149367f418f4e42954" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.510110 7780 raft_consensus.cc:3032] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.510236 7780 raft_consensus.cc:729] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.510598 7780 consensus_queue.cc:260] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.511023 7780 raft_consensus.cc:3037] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.512028 7515 raft_consensus.cc:1261] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Refusing update from remote peer e8314214a0484251964e4b83a87b6786: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:25.512826 9830 consensus_queue.cc:1035] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.513891 7514 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7ac302b18a4a41239b479f539f590b40"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.514106 7514 raft_consensus.cc:482] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.514219 7514 raft_consensus.cc:3037] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.513522 7648 raft_consensus.cc:2454] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 3.
I20240502 01:57:25.516381 7451 leader_election.cc:302] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters: 87e2815274bf41c4ace764c5e37599b5
I20240502 01:57:25.517102 9824 raft_consensus.cc:2781] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.517252 9824 raft_consensus.cc:686] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.517558 9824 consensus_queue.cc:237] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.517885 7646 raft_consensus.cc:2454] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 3.
I20240502 01:57:25.518368 7451 leader_election.cc:302] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:25.518436 7514 raft_consensus.cc:504] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.518780 9834 raft_consensus.cc:2781] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.518918 9834 raft_consensus.cc:686] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.519255 9834 consensus_queue.cc:237] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.519788 7514 leader_election.cc:288] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 2 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.519827 7780 raft_consensus.cc:2396] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d599c2145ef344fbbe26469d4331175d for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.520370 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac302b18a4a41239b479f539f590b40" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.520627 7379 raft_consensus.cc:3037] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.523646 7379 raft_consensus.cc:2454] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 2.
I20240502 01:57:25.523706 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac302b18a4a41239b479f539f590b40" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.523675 7514 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5bcb764b10d43419e58fdcfeadb2143"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.524045 7514 raft_consensus.cc:482] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.524173 7450 leader_election.cc:302] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters:
I20240502 01:57:25.524353 7514 raft_consensus.cc:3037] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.524478 9821 raft_consensus.cc:2781] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.525333 7646 raft_consensus.cc:3032] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.525506 7646 raft_consensus.cc:729] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.525820 7646 consensus_queue.cc:260] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.526358 7646 raft_consensus.cc:3037] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.532244 7514 raft_consensus.cc:504] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.533365 7646 raft_consensus.cc:2454] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 2.
I20240502 01:57:25.533452 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcb764b10d43419e58fdcfeadb2143" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.533704 7780 raft_consensus.cc:3032] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.533838 7780 raft_consensus.cc:729] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.534133 7780 consensus_queue.cc:260] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.534590 7780 raft_consensus.cc:3037] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.536823 7382 raft_consensus.cc:1261] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Refusing update from remote peer e8314214a0484251964e4b83a87b6786: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:25.538956 9662 consensus_queue.cc:1035] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.539103 7780 raft_consensus.cc:2454] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 2.
I20240502 01:57:25.539685 7448 leader_election.cc:302] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.540354 7514 leader_election.cc:288] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 2 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.541469 9823 raft_consensus.cc:2781] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.541668 9823 raft_consensus.cc:686] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [term 2 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.541847 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcb764b10d43419e58fdcfeadb2143" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.542089 7381 raft_consensus.cc:3037] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.541968 9823 consensus_queue.cc:237] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.545562 7381 raft_consensus.cc:2454] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 2.
I20240502 01:57:25.549329 7514 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1b9c54c770fd47e4b92a804de3b2bcf3"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.549714 7514 raft_consensus.cc:482] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.550086 7514 raft_consensus.cc:3037] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.554349 7514 raft_consensus.cc:504] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.558307 7228 catalog_manager.cc:5484] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 2 to 3, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 3 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } }
I20240502 01:57:25.559134 9821 raft_consensus.cc:686] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [term 2 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.559464 9821 consensus_queue.cc:237] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.563176 7514 leader_election.cc:288] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:25.564769 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9c54c770fd47e4b92a804de3b2bcf3" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.564918 7514 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f8f40cb87e054bb28ea0ab7aa9db20ff"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.565039 7780 raft_consensus.cc:3032] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.565115 7514 raft_consensus.cc:482] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.565151 7780 raft_consensus.cc:729] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.565235 7514 raft_consensus.cc:3037] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.565430 7780 consensus_queue.cc:260] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.565814 7780 raft_consensus.cc:3037] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.566653 7228 catalog_manager.cc:5484] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 2 to 3, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 3 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
I20240502 01:57:25.567210 7228 catalog_manager.cc:5484] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 1 to 2, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 2 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } }
I20240502 01:57:25.567625 7228 catalog_manager.cc:5484] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 2 to 3, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 3 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:25.568127 7228 catalog_manager.cc:5484] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 2 to 3, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 3 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:25.568231 7514 raft_consensus.cc:504] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.568521 7228 catalog_manager.cc:5484] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 2 to 3, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 3 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:25.569177 7514 leader_election.cc:288] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.569319 7780 raft_consensus.cc:2396] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d599c2145ef344fbbe26469d4331175d for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.569173 7228 catalog_manager.cc:5484] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 2 to 3, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 3 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:25.569423 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f40cb87e054bb28ea0ab7aa9db20ff" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.569666 7646 raft_consensus.cc:3032] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.569777 7646 raft_consensus.cc:729] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.569595 7228 catalog_manager.cc:5484] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 1 to 2, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 2 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:25.570045 7646 consensus_queue.cc:260] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.570031 7228 catalog_manager.cc:5484] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 1 to 2, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 2 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:25.570446 7515 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e4bb4b288a4d4fd689b750da93d2fa07"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.570541 7646 raft_consensus.cc:3037] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.570622 7515 raft_consensus.cc:482] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.570732 7515 raft_consensus.cc:3037] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.573449 7646 raft_consensus.cc:2454] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 3.
I20240502 01:57:25.573521 7515 raft_consensus.cc:504] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.573936 7451 leader_election.cc:302] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:25.574201 7515 leader_election.cc:288] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:25.574681 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4bb4b288a4d4fd689b750da93d2fa07" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.574911 7648 raft_consensus.cc:3032] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.575008 7648 raft_consensus.cc:729] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.575356 7648 consensus_queue.cc:260] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.575728 7648 raft_consensus.cc:3037] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.577348 9821 raft_consensus.cc:2781] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.577525 9821 raft_consensus.cc:686] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.577801 9821 consensus_queue.cc:237] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.581797 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f40cb87e054bb28ea0ab7aa9db20ff" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.582043 7381 raft_consensus.cc:3037] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.582239 7515 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c557be7764d456fa5d8a7883223951c"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.582443 7515 raft_consensus.cc:482] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.582571 7515 raft_consensus.cc:3037] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.585644 7515 raft_consensus.cc:504] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.586272 7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4bb4b288a4d4fd689b750da93d2fa07" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.586385 7515 leader_election.cc:288] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.586998 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c557be7764d456fa5d8a7883223951c" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.587227 7780 raft_consensus.cc:3032] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.587335 7780 raft_consensus.cc:729] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.587546 7514 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3296950eda164749ab23df9d041ff456"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.587628 7780 consensus_queue.cc:260] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.587751 7514 raft_consensus.cc:482] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.587848 7514 raft_consensus.cc:3037] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.587967 7780 raft_consensus.cc:3037] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.590643 7514 raft_consensus.cc:504] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.591043 7780 raft_consensus.cc:2396] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d599c2145ef344fbbe26469d4331175d for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.591439 7514 leader_election.cc:288] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.592140 7777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296950eda164749ab23df9d041ff456" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.591959 7235 catalog_manager.cc:5484] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 2 to 3, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 3 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:25.592334 7777 raft_consensus.cc:3032] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.592424 7777 raft_consensus.cc:729] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.592715 7777 consensus_queue.cc:260] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.593096 7777 raft_consensus.cc:3037] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.593487 7648 raft_consensus.cc:2454] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 3.
I20240502 01:57:25.593145 7514 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3c56e67fc824a8799e0218b245f52c9"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.593807 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c557be7764d456fa5d8a7883223951c" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.593943 7514 raft_consensus.cc:482] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.594058 7382 raft_consensus.cc:3037] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.594077 7514 raft_consensus.cc:3037] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.596004 7777 raft_consensus.cc:2396] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d599c2145ef344fbbe26469d4331175d for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.596944 7514 raft_consensus.cc:504] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.597095 7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296950eda164749ab23df9d041ff456" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.597330 7378 raft_consensus.cc:3037] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.598186 7451 leader_election.cc:302] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:25.598160 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c56e67fc824a8799e0218b245f52c9" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.598353 7780 raft_consensus.cc:3032] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.598446 7780 raft_consensus.cc:729] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.598513 9821 raft_consensus.cc:2781] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.598680 9821 raft_consensus.cc:686] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.598888 7780 consensus_queue.cc:260] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.598963 9821 consensus_queue.cc:237] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.599285 7780 raft_consensus.cc:3037] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.600108 7378 raft_consensus.cc:2454] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 3.
I20240502 01:57:25.600438 7514 leader_election.cc:288] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 2 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:25.601637 7515 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dffadd26fd38434eaf6cda23da9f1408"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.601903 7515 raft_consensus.cc:482] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.601984 7780 raft_consensus.cc:2454] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 2.
I20240502 01:57:25.602128 7515 raft_consensus.cc:3037] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.602531 7448 leader_election.cc:302] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:25.603008 9821 raft_consensus.cc:2781] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.603364 9821 raft_consensus.cc:686] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [term 2 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.603869 9821 consensus_queue.cc:237] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.604069 7382 raft_consensus.cc:2454] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 3.
I20240502 01:57:25.604518 7377 raft_consensus.cc:1261] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Refusing update from remote peer fb68cc03fe5644e0b4846a80d385c412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:57:25.605176 9799 consensus_queue.cc:1035] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.605648 7450 leader_election.cc:302] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters: fb68cc03fe5644e0b4846a80d385c412
I20240502 01:57:25.606411 7377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c56e67fc824a8799e0218b245f52c9" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.606652 7377 raft_consensus.cc:3037] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.609683 7377 raft_consensus.cc:2454] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 2.
I20240502 01:57:25.610085 7381 raft_consensus.cc:2454] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 3.
I20240502 01:57:25.610550 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9c54c770fd47e4b92a804de3b2bcf3" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.610759 7379 raft_consensus.cc:3037] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.605860 7515 raft_consensus.cc:504] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.614981 9842 raft_consensus.cc:2781] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.616401 7379 raft_consensus.cc:2454] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 3.
I20240502 01:57:25.615175 9842 raft_consensus.cc:686] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.617161 9842 consensus_queue.cc:237] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.618387 7450 leader_election.cc:302] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters: fb68cc03fe5644e0b4846a80d385c412
I20240502 01:57:25.619457 7450 leader_election.cc:302] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters: fb68cc03fe5644e0b4846a80d385c412
I20240502 01:57:25.619771 9832 raft_consensus.cc:2781] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.619813 9842 raft_consensus.cc:2781] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.619951 9842 raft_consensus.cc:686] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.620028 9832 raft_consensus.cc:686] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.620240 9842 consensus_queue.cc:237] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.620237 9832 consensus_queue.cc:237] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.631532 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffadd26fd38434eaf6cda23da9f1408" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.631635 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffadd26fd38434eaf6cda23da9f1408" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.631801 7780 raft_consensus.cc:3032] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.631934 7780 raft_consensus.cc:729] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.631974 7648 raft_consensus.cc:3037] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.632273 7780 consensus_queue.cc:260] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.632681 7780 raft_consensus.cc:3037] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.632781 7515 leader_election.cc:288] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:25.635736 7648 raft_consensus.cc:2454] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 3.
I20240502 01:57:25.636540 7780 raft_consensus.cc:2396] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d599c2145ef344fbbe26469d4331175d for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.636648 7451 leader_election.cc:302] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:25.637434 9832 raft_consensus.cc:2781] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.637671 9832 raft_consensus.cc:686] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.638036 9832 consensus_queue.cc:237] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.638820 7228 catalog_manager.cc:5484] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 2 to 3, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 3 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
I20240502 01:57:25.639559 7228 catalog_manager.cc:5484] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 1 to 2, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 2 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } }
I20240502 01:57:25.647806 7515 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9819568333624685aafc31d82e22f0e0"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.648281 7515 raft_consensus.cc:482] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.648583 7515 raft_consensus.cc:3037] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:25.659795 7235 catalog_manager.cc:5484] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 2 to 3, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 3 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:25.660322 7235 catalog_manager.cc:5484] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 2 to 3, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 3 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } }
I20240502 01:57:25.660702 7235 catalog_manager.cc:5484] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 2 to 3, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 3 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:25.661283 7235 catalog_manager.cc:5484] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 2 to 3, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 3 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:25.654114 7515 raft_consensus.cc:504] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.667304 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9819568333624685aafc31d82e22f0e0" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.667313 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9819568333624685aafc31d82e22f0e0" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.667567 7648 raft_consensus.cc:3037] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:25.667579 7780 raft_consensus.cc:3032] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:57:25.667690 7780 raft_consensus.cc:729] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.667973 7780 consensus_queue.cc:260] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.668315 7780 raft_consensus.cc:3037] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:25.670754 7648 raft_consensus.cc:2396] T 9819568333624685aafc31d82e22f0e0 P e8314214a0484251964e4b83a87b6786 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d599c2145ef344fbbe26469d4331175d for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.670800 7646 raft_consensus.cc:1261] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Refusing update from remote peer fb68cc03fe5644e0b4846a80d385c412: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:57:25.670970 7646 pending_rounds.cc:76] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786: Aborting all ops after (but not including) 3
I20240502 01:57:25.671099 7646 pending_rounds.cc:97] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:57:25.671285 7646 raft_consensus.cc:2866] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:57:25.671500 7515 leader_election.cc:288] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 4 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:25.672137 7780 raft_consensus.cc:2396] T 9819568333624685aafc31d82e22f0e0 P fb68cc03fe5644e0b4846a80d385c412 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d599c2145ef344fbbe26469d4331175d for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.672346 9799 consensus_queue.cc:1035] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.672755 7448 leader_election.cc:302] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d599c2145ef344fbbe26469d4331175d; no voters: e8314214a0484251964e4b83a87b6786, fb68cc03fe5644e0b4846a80d385c412
I20240502 01:57:25.673799 9832 raft_consensus.cc:2726] T 9819568333624685aafc31d82e22f0e0 P d599c2145ef344fbbe26469d4331175d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:57:25.673556 7515 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "04426a27d50548b9a3eb0e3eb09a5bcc"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.674618 7515 raft_consensus.cc:482] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.674777 7515 raft_consensus.cc:3037] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.678694 7515 raft_consensus.cc:504] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.679653 7515 leader_election.cc:288] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 2 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:25.679973 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04426a27d50548b9a3eb0e3eb09a5bcc" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.679977 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04426a27d50548b9a3eb0e3eb09a5bcc" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.680191 7646 raft_consensus.cc:3032] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.680210 7382 raft_consensus.cc:3037] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.680298 7646 raft_consensus.cc:729] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.680584 7646 consensus_queue.cc:260] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.681030 7646 raft_consensus.cc:3037] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.683166 7382 raft_consensus.cc:2454] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 2.
I20240502 01:57:25.683785 7646 raft_consensus.cc:2454] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 2.
I20240502 01:57:25.683914 7450 leader_election.cc:302] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters:
I20240502 01:57:25.684583 9832 raft_consensus.cc:2781] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.684746 9832 raft_consensus.cc:686] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [term 2 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.685256 9832 consensus_queue.cc:237] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.686044 7515 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d727fc7ec2054b46aafb44ccf4bd0ad8"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.686286 7515 raft_consensus.cc:482] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.686467 7515 raft_consensus.cc:3037] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.689795 7515 raft_consensus.cc:504] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.690428 7515 leader_election.cc:288] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 2 election: Requested vote from peers 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:25.690964 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d727fc7ec2054b46aafb44ccf4bd0ad8" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.691174 7382 raft_consensus.cc:3037] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.691665 7235 catalog_manager.cc:5484] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 1 to 2, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 2 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
I20240502 01:57:25.693048 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d727fc7ec2054b46aafb44ccf4bd0ad8" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.693274 7646 raft_consensus.cc:3032] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:25.693377 7646 raft_consensus.cc:729] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8314214a0484251964e4b83a87b6786, State: Running, Role: LEADER
I20240502 01:57:25.693631 7382 raft_consensus.cc:2454] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 2.
I20240502 01:57:25.693646 7646 consensus_queue.cc:260] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.694118 7646 raft_consensus.cc:3037] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:25.694223 7450 leader_election.cc:302] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters:
I20240502 01:57:25.694255 7515 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7a14895a05814947bcec5111f41fdfb9"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.694439 7515 raft_consensus.cc:482] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.694586 7515 raft_consensus.cc:3037] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.694754 9832 raft_consensus.cc:2781] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:25.695086 9832 raft_consensus.cc:686] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [term 2 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.695436 9832 consensus_queue.cc:237] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.697013 7646 raft_consensus.cc:2454] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 2.
I20240502 01:57:25.698199 7515 raft_consensus.cc:504] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.699638 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a14895a05814947bcec5111f41fdfb9" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.699927 7780 raft_consensus.cc:3032] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.700062 7780 raft_consensus.cc:729] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.700407 7780 consensus_queue.cc:260] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.700779 7780 raft_consensus.cc:3037] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.701179 7382 raft_consensus.cc:1261] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Refusing update from remote peer fb68cc03fe5644e0b4846a80d385c412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:25.704411 7780 raft_consensus.cc:2396] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d599c2145ef344fbbe26469d4331175d for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.705196 7515 leader_election.cc:288] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193:40913)
I20240502 01:57:25.705737 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a14895a05814947bcec5111f41fdfb9" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
I20240502 01:57:25.705998 7382 raft_consensus.cc:3037] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.708338 7382 raft_consensus.cc:2454] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 3.
I20240502 01:57:25.708787 7450 leader_election.cc:302] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters: fb68cc03fe5644e0b4846a80d385c412
I20240502 01:57:25.709184 9832 raft_consensus.cc:2781] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.709357 9832 raft_consensus.cc:686] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.709535 7515 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26522a1c09bf44aca91d0410a7459b26"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.709764 7515 raft_consensus.cc:482] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.709722 9832 consensus_queue.cc:237] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.709877 7515 raft_consensus.cc:3037] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.712208 7382 raft_consensus.cc:1261] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Refusing update from remote peer fb68cc03fe5644e0b4846a80d385c412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:25.712801 7515 raft_consensus.cc:504] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.712814 9799 consensus_queue.cc:1035] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.713686 7515 leader_election.cc:288] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:25.713999 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26522a1c09bf44aca91d0410a7459b26" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.714236 7646 raft_consensus.cc:3037] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.714231 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26522a1c09bf44aca91d0410a7459b26" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.714435 7780 raft_consensus.cc:3032] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.714555 7780 raft_consensus.cc:729] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.714813 7780 consensus_queue.cc:260] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.715140 7780 raft_consensus.cc:3037] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.715222 7515 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0718691ffdc04f719ad7df57987df68c"
dest_uuid: "d599c2145ef344fbbe26469d4331175d"
from {username='slave'} at 127.0.0.1:46816
I20240502 01:57:25.715453 7515 raft_consensus.cc:482] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.715596 7515 raft_consensus.cc:3037] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.717206 7646 raft_consensus.cc:2454] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 3.
I20240502 01:57:25.717638 7451 leader_election.cc:302] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:25.717970 7780 raft_consensus.cc:2396] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d599c2145ef344fbbe26469d4331175d for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.720378 7515 raft_consensus.cc:504] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.721293 7515 leader_election.cc:288] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:25.721581 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0718691ffdc04f719ad7df57987df68c" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.721823 7780 raft_consensus.cc:3032] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.721954 7780 raft_consensus.cc:729] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.721530 9832 raft_consensus.cc:2781] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.722235 9832 raft_consensus.cc:686] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.722250 7780 consensus_queue.cc:260] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.722585 7780 raft_consensus.cc:3037] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.722512 9832 consensus_queue.cc:237] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.722707 7382 raft_consensus.cc:1261] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Refusing update from remote peer fb68cc03fe5644e0b4846a80d385c412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:57:25.723243 9802 consensus_queue.cc:1035] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.725723 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0718691ffdc04f719ad7df57987df68c" candidate_uuid: "d599c2145ef344fbbe26469d4331175d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.726294 7647 raft_consensus.cc:1261] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Refusing update from remote peer fb68cc03fe5644e0b4846a80d385c412: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:57:25.726413 7647 pending_rounds.cc:76] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786: Aborting all ops after (but not including) 3
I20240502 01:57:25.726454 7648 raft_consensus.cc:1261] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Refusing update from remote peer fb68cc03fe5644e0b4846a80d385c412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:25.726521 7647 pending_rounds.cc:97] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:57:25.726595 7647 raft_consensus.cc:2866] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:57:25.728205 9803 consensus_queue.cc:1035] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.728243 9802 consensus_queue.cc:1035] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.728704 7646 raft_consensus.cc:3037] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.729271 7647 raft_consensus.cc:1261] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Refusing update from remote peer fb68cc03fe5644e0b4846a80d385c412: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:57:25.729385 7647 pending_rounds.cc:76] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786: Aborting all ops after (but not including) 3
I20240502 01:57:25.729465 7647 pending_rounds.cc:97] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:57:25.729532 7647 raft_consensus.cc:2866] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:57:25.730309 7780 raft_consensus.cc:2396] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d599c2145ef344fbbe26469d4331175d for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.731333 7646 raft_consensus.cc:2454] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d599c2145ef344fbbe26469d4331175d in term 3.
I20240502 01:57:25.733692 7382 raft_consensus.cc:1261] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Refusing update from remote peer fb68cc03fe5644e0b4846a80d385c412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:25.733474 7228 catalog_manager.cc:5484] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 1 to 2, leader changed from e8314214a0484251964e4b83a87b6786 (127.14.212.195) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 2 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
I20240502 01:57:25.734267 9802 consensus_queue.cc:1035] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.735859 7379 raft_consensus.cc:1261] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Refusing update from remote peer fb68cc03fe5644e0b4846a80d385c412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:57:25.737342 9802 consensus_queue.cc:1035] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.739013 7451 leader_election.cc:302] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786; no voters: fb68cc03fe5644e0b4846a80d385c412
I20240502 01:57:25.741034 9802 consensus_queue.cc:1035] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.742514 7229 catalog_manager.cc:5484] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 2 to 3, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 3 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:25.743062 7229 catalog_manager.cc:5484] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 2 to 3, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 3 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:25.744053 9832 raft_consensus.cc:2781] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.744307 9832 raft_consensus.cc:686] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Leader. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.744616 9832 consensus_queue.cc:237] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:25.747503 7381 raft_consensus.cc:1261] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Refusing update from remote peer fb68cc03fe5644e0b4846a80d385c412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:57:25.748203 9799 consensus_queue.cc:1035] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.761682 7646 raft_consensus.cc:1261] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Refusing update from remote peer fb68cc03fe5644e0b4846a80d385c412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:25.762346 9802 consensus_queue.cc:1035] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.768246 7647 raft_consensus.cc:1261] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Refusing update from remote peer fb68cc03fe5644e0b4846a80d385c412: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:57:25.768397 7647 pending_rounds.cc:76] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786: Aborting all ops after (but not including) 3
I20240502 01:57:25.768487 7647 pending_rounds.cc:97] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:57:25.768586 7647 raft_consensus.cc:2866] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:57:25.770641 9799 consensus_queue.cc:1035] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.773468 7229 catalog_manager.cc:5484] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d reported cstate change: term changed from 2 to 3, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to d599c2145ef344fbbe26469d4331175d (127.14.212.194). New cstate: current_term: 3 leader_uuid: "d599c2145ef344fbbe26469d4331175d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
I20240502 01:57:25.810953 7648 raft_consensus.cc:1261] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Refusing update from remote peer fb68cc03fe5644e0b4846a80d385c412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:25.811635 9799 consensus_queue.cc:1035] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.819133 7381 raft_consensus.cc:1261] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Refusing update from remote peer fb68cc03fe5644e0b4846a80d385c412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:25.820562 9802 consensus_queue.cc:1035] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.857564 7381 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b443becac180479abb1071e4c829fbcb"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:25.857936 7381 raft_consensus.cc:482] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.858083 7381 raft_consensus.cc:3037] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:25.861011 7381 raft_consensus.cc:504] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.862412 7381 leader_election.cc:288] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:25.862761 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b443becac180479abb1071e4c829fbcb" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.863003 7648 raft_consensus.cc:3037] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:25.865985 7648 raft_consensus.cc:2454] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:25.866742 7314 leader_election.cc:302] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:25.866876 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b443becac180479abb1071e4c829fbcb" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.867097 7780 raft_consensus.cc:3032] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:57:25.867203 7780 raft_consensus.cc:729] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:25.867194 9505 raft_consensus.cc:2781] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:57:25.867493 7780 consensus_queue.cc:260] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.867913 7780 raft_consensus.cc:3037] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:25.870133 7381 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f5b90bdde42e4a9b98188f1a197ed835"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:25.870360 7381 raft_consensus.cc:482] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.870512 7381 raft_consensus.cc:3037] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.872453 7780 raft_consensus.cc:2454] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:25.874156 7381 raft_consensus.cc:504] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.875633 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b90bdde42e4a9b98188f1a197ed835" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:25.875995 7515 raft_consensus.cc:2379] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Leader election vote request: Denying vote to candidate 87e2815274bf41c4ace764c5e37599b5 in current term 3: Already voted for candidate d599c2145ef344fbbe26469d4331175d in this term.
I20240502 01:57:25.877354 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b90bdde42e4a9b98188f1a197ed835" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.877681 7648 raft_consensus.cc:2379] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 87e2815274bf41c4ace764c5e37599b5 in current term 3: Already voted for candidate d599c2145ef344fbbe26469d4331175d in this term.
I20240502 01:57:25.877768 9505 raft_consensus.cc:686] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [term 4 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:25.878268 7314 leader_election.cc:302] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5; no voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786
I20240502 01:57:25.878235 9505 consensus_queue.cc:237] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:25.879067 9900 raft_consensus.cc:2726] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 01:57:25.879283 7381 leader_election.cc:288] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:25.881520 7381 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e43e7e31842d47c59425ca63bc7182da"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:25.881769 7381 raft_consensus.cc:482] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.881872 7381 raft_consensus.cc:3037] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.881738 7235 catalog_manager.cc:5484] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 3 to 4, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 4 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:25.884982 7381 raft_consensus.cc:504] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.885979 7381 leader_election.cc:288] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:25.886341 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43e7e31842d47c59425ca63bc7182da" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:25.886365 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43e7e31842d47c59425ca63bc7182da" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:25.886555 7780 raft_consensus.cc:3037] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.886622 7515 raft_consensus.cc:3032] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:25.886732 7515 raft_consensus.cc:729] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.887089 7515 consensus_queue.cc:260] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.887431 7515 raft_consensus.cc:3037] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:25.891443 7515 raft_consensus.cc:2396] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 87e2815274bf41c4ace764c5e37599b5 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.893114 7648 raft_consensus.cc:1261] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:57:25.894241 7382 raft_consensus.cc:1261] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:57:25.894798 7780 raft_consensus.cc:2454] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 3.
I20240502 01:57:25.895162 9832 consensus_queue.cc:1035] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.895327 7311 leader_election.cc:302] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters: d599c2145ef344fbbe26469d4331175d
I20240502 01:57:25.895920 9901 raft_consensus.cc:2781] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:25.896131 9901 raft_consensus.cc:686] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 [term 3 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:25.896512 9901 consensus_queue.cc:237] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:25.897496 7382 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4112f382c8e24680ae2350d9d5d8af54"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:25.897701 7382 raft_consensus.cc:482] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.897814 7382 raft_consensus.cc:3037] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:25.898450 9832 consensus_queue.cc:1035] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.902225 7382 raft_consensus.cc:504] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.902670 7377 raft_consensus.cc:1261] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Refusing update from remote peer fb68cc03fe5644e0b4846a80d385c412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:25.902794 7379 raft_consensus.cc:1226] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Rejecting Update request from peer d599c2145ef344fbbe26469d4331175d for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20240502 01:57:25.903491 9802 consensus_queue.cc:1035] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.903892 9832 consensus_queue.cc:1046] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.904284 9832 raft_consensus.cc:3032] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:57:25.904402 9832 raft_consensus.cc:729] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:25.905871 7646 raft_consensus.cc:1261] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:57:25.906024 7646 pending_rounds.cc:76] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786: Aborting all ops after (but not including) 3
I20240502 01:57:25.906097 7646 pending_rounds.cc:97] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:57:25.904767 9832 consensus_queue.cc:260] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:25.906219 7646 raft_consensus.cc:2866] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:57:25.906491 9832 raft_consensus.cc:3037] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:25.907048 9842 consensus_queue.cc:1035] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.911473 7648 raft_consensus.cc:1261] T 15e07df96bd94ef8b505a7af2ac88db4 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:57:25.912806 9842 consensus_queue.cc:1035] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.916754 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4112f382c8e24680ae2350d9d5d8af54" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:25.917047 7646 raft_consensus.cc:3037] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:25.920203 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4112f382c8e24680ae2350d9d5d8af54" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:25.920598 7382 leader_election.cc:288] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:25.921816 7382 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "29dce0c6f40a4358ab8855cf0c709862"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:25.922444 7780 raft_consensus.cc:1261] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:57:25.923108 7382 raft_consensus.cc:482] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:25.923144 7780 pending_rounds.cc:76] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412: Aborting all ops after (but not including) 3
I20240502 01:57:25.923296 7382 raft_consensus.cc:3037] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:25.923516 7780 pending_rounds.cc:97] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:57:25.923970 7780 raft_consensus.cc:2866] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:57:25.925130 9372 consensus_queue.cc:1035] T 6f65accc9dfb4bd2b8cf82cb2d3e171c P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.927847 7515 raft_consensus.cc:2396] T 4112f382c8e24680ae2350d9d5d8af54 P d599c2145ef344fbbe26469d4331175d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 87e2815274bf41c4ace764c5e37599b5 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.930734 9770 heartbeater.cc:503] Master 127.14.212.254:39055 was elected leader, sending a full tablet report...
I20240502 01:57:25.932381 7646 raft_consensus.cc:2396] T 4112f382c8e24680ae2350d9d5d8af54 P e8314214a0484251964e4b83a87b6786 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 87e2815274bf41c4ace764c5e37599b5 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:25.932623 7235 catalog_manager.cc:5484] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 2 to 3, leader changed from d599c2145ef344fbbe26469d4331175d (127.14.212.194) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 3 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } }
I20240502 01:57:25.932924 7314 leader_election.cc:302] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5; no voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786
I20240502 01:57:25.941905 7780 raft_consensus.cc:1261] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:57:25.942085 7780 pending_rounds.cc:76] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412: Aborting all ops after (but not including) 3
I20240502 01:57:25.942179 7780 pending_rounds.cc:97] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:57:25.942287 7780 raft_consensus.cc:2866] T 15e07df96bd94ef8b505a7af2ac88db4 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:57:25.943017 9832 consensus_queue.cc:1035] T 15e07df96bd94ef8b505a7af2ac88db4 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.955457 7377 raft_consensus.cc:1261] T f5b90bdde42e4a9b98188f1a197ed835 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 01:57:25.956326 9831 consensus_queue.cc:1035] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:57:25.957990 9505 raft_consensus.cc:2726] T 4112f382c8e24680ae2350d9d5d8af54 P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:57:25.961174 7646 raft_consensus.cc:1261] T f5b90bdde42e4a9b98188f1a197ed835 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 01:57:25.961830 9372 consensus_queue.cc:1035] T f5b90bdde42e4a9b98188f1a197ed835 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:57:25.971750 7777 raft_consensus.cc:1261] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:25.972479 7646 raft_consensus.cc:1261] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:25.972568 9372 consensus_queue.cc:1035] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.973980 9372 consensus_queue.cc:1035] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.976637 7377 raft_consensus.cc:1261] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 01:57:25.978224 7648 raft_consensus.cc:1261] T 71ee1cc0eacc45149367f418f4e42954 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:57:25.979144 9372 consensus_queue.cc:1035] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.982077 7645 raft_consensus.cc:1261] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Refusing update from remote peer fb68cc03fe5644e0b4846a80d385c412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:25.982874 9372 consensus_queue.cc:1035] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:57:25.985342 9803 consensus_queue.cc:1035] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.989290 7377 raft_consensus.cc:1261] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:25.990983 9823 consensus_queue.cc:1035] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.994067 7377 raft_consensus.cc:1261] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:25.995466 9823 consensus_queue.cc:1035] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:25.995918 7646 raft_consensus.cc:1261] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 01:57:25.997470 9821 consensus_queue.cc:1035] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:57:26.001562 7777 raft_consensus.cc:1261] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:57:26.001736 7777 pending_rounds.cc:76] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412: Aborting all ops after (but not including) 3
I20240502 01:57:26.001818 7777 pending_rounds.cc:97] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:57:26.002130 7777 raft_consensus.cc:2866] T 71ee1cc0eacc45149367f418f4e42954 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:57:26.007534 9372 consensus_queue.cc:1035] T 71ee1cc0eacc45149367f418f4e42954 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.015025 7382 raft_consensus.cc:504] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:26.025552 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29dce0c6f40a4358ab8855cf0c709862" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:26.025841 7515 raft_consensus.cc:3032] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:57:26.025920 7515 raft_consensus.cc:729] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:26.026196 7515 consensus_queue.cc:260] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:26.026504 7515 raft_consensus.cc:3037] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.027910 7378 raft_consensus.cc:1261] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:57:26.029368 9831 consensus_queue.cc:1035] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.030972 7515 raft_consensus.cc:2396] T 29dce0c6f40a4358ab8855cf0c709862 P d599c2145ef344fbbe26469d4331175d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 87e2815274bf41c4ace764c5e37599b5 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:26.031178 7777 raft_consensus.cc:1261] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:57:26.031352 7777 pending_rounds.cc:76] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412: Aborting all ops after (but not including) 3
I20240502 01:57:26.031432 7777 pending_rounds.cc:97] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:57:26.031591 7777 raft_consensus.cc:2866] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:57:26.032583 9831 consensus_queue.cc:1035] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.038022 7382 leader_election.cc:288] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:26.040401 7377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d727fc7ec2054b46aafb44ccf4bd0ad8"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:26.040661 7377 raft_consensus.cc:482] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:26.040771 7377 raft_consensus.cc:3037] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.041345 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29dce0c6f40a4358ab8855cf0c709862" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:26.041761 7646 raft_consensus.cc:3037] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.044195 7377 raft_consensus.cc:504] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:26.045557 7377 leader_election.cc:288] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:26.046265 7645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d727fc7ec2054b46aafb44ccf4bd0ad8" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:26.046496 7645 raft_consensus.cc:3037] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.048082 7777 raft_consensus.cc:1261] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:57:26.048259 7777 pending_rounds.cc:76] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412: Aborting all ops after (but not including) 3
I20240502 01:57:26.048353 7777 pending_rounds.cc:97] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:57:26.048460 7777 raft_consensus.cc:2866] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:57:26.049480 9821 consensus_queue.cc:1035] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.050377 7645 raft_consensus.cc:2454] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 3.
I20240502 01:57:26.050944 7314 leader_election.cc:302] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:26.051380 9505 raft_consensus.cc:2781] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:26.051683 9505 raft_consensus.cc:686] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 [term 3 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:26.052138 9505 consensus_queue.cc:237] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:26.052953 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d727fc7ec2054b46aafb44ccf4bd0ad8" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:26.053200 7515 raft_consensus.cc:3032] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:26.053283 7515 raft_consensus.cc:729] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:26.053556 7515 consensus_queue.cc:260] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:26.053786 7515 raft_consensus.cc:3037] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.056396 7515 raft_consensus.cc:2396] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 87e2815274bf41c4ace764c5e37599b5 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:26.058058 7777 raft_consensus.cc:1261] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:57:26.059422 7645 raft_consensus.cc:1261] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 01:57:26.060173 9824 consensus_queue.cc:1035] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:57:26.062379 7777 pending_rounds.cc:76] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412: Aborting all ops after (but not including) 3
I20240502 01:57:26.062507 7777 pending_rounds.cc:97] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:57:26.062623 7777 raft_consensus.cc:2866] T dffadd26fd38434eaf6cda23da9f1408 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:57:26.063390 9372 consensus_queue.cc:1035] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.063894 7378 raft_consensus.cc:3037] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.072999 7645 raft_consensus.cc:1261] T dffadd26fd38434eaf6cda23da9f1408 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:57:26.073081 7377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c93a84c310742b999bbb053d687639f"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:26.073307 7377 raft_consensus.cc:482] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:26.073479 7377 raft_consensus.cc:3037] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.075151 9832 consensus_queue.cc:1035] T dffadd26fd38434eaf6cda23da9f1408 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.077651 7232 catalog_manager.cc:5484] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 2 to 3, leader changed from d599c2145ef344fbbe26469d4331175d (127.14.212.194) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 3 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
I20240502 01:57:26.080262 7378 raft_consensus.cc:1261] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 01:57:26.080965 7646 raft_consensus.cc:2396] T 29dce0c6f40a4358ab8855cf0c709862 P e8314214a0484251964e4b83a87b6786 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 87e2815274bf41c4ace764c5e37599b5 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:57:26.081521 7314 leader_election.cc:302] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5; no voters: d599c2145ef344fbbe26469d4331175d, e8314214a0484251964e4b83a87b6786
I20240502 01:57:26.083627 7377 raft_consensus.cc:504] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:26.023625 7381 raft_consensus.cc:1226] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Rejecting Update request from peer d599c2145ef344fbbe26469d4331175d for earlier term 3. Current term is 4. Ops: []
I20240502 01:57:26.084590 9505 raft_consensus.cc:2726] T 29dce0c6f40a4358ab8855cf0c709862 P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:57:26.085896 9832 consensus_queue.cc:1035] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:57:26.086951 7777 raft_consensus.cc:1261] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:57:26.087092 7777 pending_rounds.cc:76] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412: Aborting all ops after (but not including) 3
I20240502 01:57:26.088372 7777 pending_rounds.cc:97] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:57:26.088514 7777 raft_consensus.cc:2866] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:57:26.089222 9832 consensus_queue.cc:1035] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.086019 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c93a84c310742b999bbb053d687639f" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:26.091008 7646 raft_consensus.cc:3037] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.091535 7381 raft_consensus.cc:1261] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:57:26.096251 7378 raft_consensus.cc:1261] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:57:26.096400 9832 consensus_queue.cc:1035] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.097373 7382 raft_consensus.cc:1261] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:57:26.098167 7777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c93a84c310742b999bbb053d687639f" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:26.098435 7777 raft_consensus.cc:3032] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:26.098533 7777 raft_consensus.cc:729] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:26.098834 7777 consensus_queue.cc:260] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:26.099195 7777 raft_consensus.cc:3037] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.102417 7777 raft_consensus.cc:2454] T 9c93a84c310742b999bbb053d687639f P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 3.
I20240502 01:57:26.102990 7311 leader_election.cc:302] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:26.103374 9505 raft_consensus.cc:2781] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:26.103551 9505 raft_consensus.cc:686] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5 [term 3 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:26.103884 9505 consensus_queue.cc:237] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:26.105573 9824 consensus_queue.cc:1035] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.106065 9824 consensus_queue.cc:1035] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.107568 7377 leader_election.cc:288] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:26.109054 7378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7a14895a05814947bcec5111f41fdfb9"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:26.111243 7646 raft_consensus.cc:2454] T 9c93a84c310742b999bbb053d687639f P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 3.
I20240502 01:57:26.113302 7232 catalog_manager.cc:5484] T 9c93a84c310742b999bbb053d687639f P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 2 to 3, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 3 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:26.115531 7377 raft_consensus.cc:1261] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:26.121443 7378 raft_consensus.cc:482] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:26.121649 7378 raft_consensus.cc:3037] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.124186 7378 raft_consensus.cc:504] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.125769 7777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a14895a05814947bcec5111f41fdfb9" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:26.126044 7777 raft_consensus.cc:3037] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.128491 7777 raft_consensus.cc:2454] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:26.129129 7311 leader_election.cc:302] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:26.129491 9505 raft_consensus.cc:2781] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:57:26.129693 9505 raft_consensus.cc:686] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 4 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:26.130028 9505 consensus_queue.cc:237] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.130329 7378 leader_election.cc:288] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:26.131279 9832 consensus_queue.cc:1035] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.136603 7382 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1b9c54c770fd47e4b92a804de3b2bcf3"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:26.137042 7777 raft_consensus.cc:1226] T 7a14895a05814947bcec5111f41fdfb9 P fb68cc03fe5644e0b4846a80d385c412 [term 4 FOLLOWER]: Rejecting Update request from peer d599c2145ef344fbbe26469d4331175d for earlier term 3. Current term is 4. Ops: []
I20240502 01:57:26.137773 9824 consensus_queue.cc:1046] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.138037 9824 consensus_queue.cc:1225] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [LEADER]: Peer fb68cc03fe5644e0b4846a80d385c412 log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240502 01:57:26.138118 9570 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:26.138196 9824 raft_consensus.cc:3032] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:57:26.138279 9824 raft_consensus.cc:729] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:26.138597 9824 consensus_queue.cc:260] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.138981 9824 raft_consensus.cc:3037] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.144052 7379 raft_consensus.cc:1226] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 [term 4 LEADER]: Rejecting Update request from peer d599c2145ef344fbbe26469d4331175d for earlier term 3. Current term is 4. Ops: []
I20240502 01:57:26.144184 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a14895a05814947bcec5111f41fdfb9" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:26.148138 7515 raft_consensus.cc:2454] T 7a14895a05814947bcec5111f41fdfb9 P d599c2145ef344fbbe26469d4331175d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:26.151602 9434 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:26.170138 7381 raft_consensus.cc:1261] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:26.172333 9372 consensus_queue.cc:1035] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.183773 7777 raft_consensus.cc:1261] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:26.184566 9372 consensus_queue.cc:1035] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.190961 9300 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:26.196791 7382 raft_consensus.cc:482] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:26.197086 7382 raft_consensus.cc:3037] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.205098 7382 raft_consensus.cc:504] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:26.206519 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9c54c770fd47e4b92a804de3b2bcf3" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:26.206792 7515 raft_consensus.cc:3032] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:57:26.206876 7515 raft_consensus.cc:729] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:26.207172 7515 consensus_queue.cc:260] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:26.207623 7515 raft_consensus.cc:3037] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.211927 7646 raft_consensus.cc:1261] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:26.212761 9832 consensus_queue.cc:1035] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.213117 7232 catalog_manager.cc:5484] T 7a14895a05814947bcec5111f41fdfb9 P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 3 to 4, leader changed from d599c2145ef344fbbe26469d4331175d (127.14.212.194) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 4 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:26.213704 7515 raft_consensus.cc:2454] T 1b9c54c770fd47e4b92a804de3b2bcf3 P d599c2145ef344fbbe26469d4331175d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:26.216256 7645 raft_consensus.cc:1261] T 26522a1c09bf44aca91d0410a7459b26 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:57:26.216852 9832 consensus_queue.cc:1035] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.219244 7313 leader_election.cc:302] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters:
I20240502 01:57:26.219689 9505 raft_consensus.cc:2781] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:57:26.219883 9505 raft_consensus.cc:686] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [term 4 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:26.220238 9505 consensus_queue.cc:237] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:26.221488 7382 leader_election.cc:288] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:26.222824 7777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9c54c770fd47e4b92a804de3b2bcf3" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:26.223197 7777 raft_consensus.cc:3037] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.226414 7777 raft_consensus.cc:2454] T 1b9c54c770fd47e4b92a804de3b2bcf3 P fb68cc03fe5644e0b4846a80d385c412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:26.226706 7378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c557be7764d456fa5d8a7883223951c"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:26.227015 7378 raft_consensus.cc:482] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:26.227228 7378 raft_consensus.cc:3037] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.230394 7378 raft_consensus.cc:504] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.232836 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c557be7764d456fa5d8a7883223951c" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:26.233309 7515 raft_consensus.cc:3032] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:57:26.233446 7515 raft_consensus.cc:729] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:26.233836 7515 consensus_queue.cc:260] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.234328 7515 raft_consensus.cc:3037] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.237983 7515 raft_consensus.cc:2454] T 8c557be7764d456fa5d8a7883223951c P d599c2145ef344fbbe26469d4331175d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:26.238158 9705 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:26.238735 7313 leader_election.cc:302] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters:
I20240502 01:57:26.239233 9505 raft_consensus.cc:2781] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:57:26.239465 9505 raft_consensus.cc:686] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [term 4 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:26.239742 9505 consensus_queue.cc:237] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.242857 7232 catalog_manager.cc:5484] T 1b9c54c770fd47e4b92a804de3b2bcf3 P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 3 to 4, leader changed from d599c2145ef344fbbe26469d4331175d (127.14.212.194) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 4 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:26.231887 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c557be7764d456fa5d8a7883223951c" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:26.248346 7780 raft_consensus.cc:3037] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.251850 7780 raft_consensus.cc:2454] T 8c557be7764d456fa5d8a7883223951c P fb68cc03fe5644e0b4846a80d385c412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:26.253749 7232 catalog_manager.cc:5484] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 3 to 4, leader changed from d599c2145ef344fbbe26469d4331175d (127.14.212.194) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 4 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:26.257684 7648 raft_consensus.cc:1261] T b443becac180479abb1071e4c829fbcb P e8314214a0484251964e4b83a87b6786 [term 4 FOLLOWER]: Refusing update from remote peer 87e2815274bf41c4ace764c5e37599b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 01:57:26.258380 9505 consensus_queue.cc:1035] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:57:26.264633 7777 raft_consensus.cc:1261] T b443becac180479abb1071e4c829fbcb P fb68cc03fe5644e0b4846a80d385c412 [term 4 FOLLOWER]: Refusing update from remote peer 87e2815274bf41c4ace764c5e37599b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 01:57:26.265676 9900 consensus_queue.cc:1035] T b443becac180479abb1071e4c829fbcb P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:57:26.271446 7378 leader_election.cc:288] T 8c557be7764d456fa5d8a7883223951c P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:26.273242 7360 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:57:26.273525 7378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1d6946feb1c14a1eaacad72be52108f3"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:26.273770 7378 raft_consensus.cc:482] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:26.273929 7378 raft_consensus.cc:3037] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.275529 7494 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:57:26.295094 7646 raft_consensus.cc:1261] T 0718691ffdc04f719ad7df57987df68c P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:57:26.296048 9832 consensus_queue.cc:1035] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.296986 7626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:26.300547 7777 raft_consensus.cc:1261] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:57:26.300693 7777 pending_rounds.cc:76] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412: Aborting all ops after (but not including) 3
I20240502 01:57:26.300767 7777 pending_rounds.cc:97] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:57:26.300850 7777 raft_consensus.cc:2866] T 26522a1c09bf44aca91d0410a7459b26 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:57:26.300947 7378 raft_consensus.cc:504] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:26.301633 9372 consensus_queue.cc:1035] T 26522a1c09bf44aca91d0410a7459b26 P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.302173 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6946feb1c14a1eaacad72be52108f3" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:26.302430 7644 raft_consensus.cc:3037] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.307775 7644 raft_consensus.cc:2454] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 3.
I20240502 01:57:26.308130 7645 raft_consensus.cc:1226] T 1d6946feb1c14a1eaacad72be52108f3 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Rejecting Update request from peer fb68cc03fe5644e0b4846a80d385c412 for earlier term 2. Current term is 3. Ops: []
I20240502 01:57:26.308344 7314 leader_election.cc:302] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:26.309065 9900 raft_consensus.cc:2781] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:26.309281 9900 raft_consensus.cc:686] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5 [term 3 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:26.309734 9900 consensus_queue.cc:237] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:26.313704 9802 consensus_queue.cc:1046] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:57:26.314186 9802 raft_consensus.cc:3032] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:26.314291 9802 raft_consensus.cc:729] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:26.314646 9802 consensus_queue.cc:260] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:26.315158 9802 raft_consensus.cc:3037] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.315834 7378 leader_election.cc:288] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:26.318245 7382 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3ac865cf3c044ac8ae3b0383b0b73c0"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:26.318502 7382 raft_consensus.cc:482] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:26.318604 7382 raft_consensus.cc:3037] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.321188 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6946feb1c14a1eaacad72be52108f3" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:26.321573 7382 raft_consensus.cc:504] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:26.323447 7645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ac865cf3c044ac8ae3b0383b0b73c0" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:26.323673 7645 raft_consensus.cc:3037] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.323689 7229 catalog_manager.cc:5484] T 1d6946feb1c14a1eaacad72be52108f3 P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 2 to 3, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 3 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:26.324901 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ac865cf3c044ac8ae3b0383b0b73c0" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:26.325150 7779 raft_consensus.cc:3032] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:57:26.325232 7779 raft_consensus.cc:729] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:26.325512 7779 consensus_queue.cc:260] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:26.325935 7779 raft_consensus.cc:3037] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.326673 7645 raft_consensus.cc:2454] T d3ac865cf3c044ac8ae3b0383b0b73c0 P e8314214a0484251964e4b83a87b6786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:26.327284 7314 leader_election.cc:302] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:26.327663 9505 raft_consensus.cc:2781] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:57:26.327775 7382 leader_election.cc:288] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:26.327860 9505 raft_consensus.cc:686] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [term 4 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:26.328142 9505 consensus_queue.cc:237] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:26.329453 7382 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f8f40cb87e054bb28ea0ab7aa9db20ff"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:26.329656 7382 raft_consensus.cc:482] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:26.329773 7382 raft_consensus.cc:3037] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.331815 7779 raft_consensus.cc:2454] T d3ac865cf3c044ac8ae3b0383b0b73c0 P fb68cc03fe5644e0b4846a80d385c412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:26.332979 7382 raft_consensus.cc:504] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.334139 7645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f40cb87e054bb28ea0ab7aa9db20ff" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:26.334440 7645 raft_consensus.cc:3037] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.337507 7645 raft_consensus.cc:2454] T f8f40cb87e054bb28ea0ab7aa9db20ff P e8314214a0484251964e4b83a87b6786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:26.338116 7314 leader_election.cc:302] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:26.338728 7382 leader_election.cc:288] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:26.338971 9900 raft_consensus.cc:2781] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:57:26.339164 9900 raft_consensus.cc:686] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [term 4 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:26.339319 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f40cb87e054bb28ea0ab7aa9db20ff" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:26.339592 7515 raft_consensus.cc:3032] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:57:26.339457 9900 consensus_queue.cc:237] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.339691 7515 raft_consensus.cc:729] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:26.339829 7382 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c94f99eccbe34c9ebf3a375e9230994d"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:26.340050 7382 raft_consensus.cc:482] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:26.339977 7515 consensus_queue.cc:260] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.340157 7382 raft_consensus.cc:3037] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.340375 7515 raft_consensus.cc:3037] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.343461 7515 raft_consensus.cc:2454] T f8f40cb87e054bb28ea0ab7aa9db20ff P d599c2145ef344fbbe26469d4331175d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:26.344611 7780 raft_consensus.cc:2454] T 1d6946feb1c14a1eaacad72be52108f3 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 3.
I20240502 01:57:26.346186 7382 raft_consensus.cc:504] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:26.347530 7645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94f99eccbe34c9ebf3a375e9230994d" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:26.347808 7645 raft_consensus.cc:3037] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.350813 7645 raft_consensus.cc:2454] T c94f99eccbe34c9ebf3a375e9230994d P e8314214a0484251964e4b83a87b6786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:26.351269 7314 leader_election.cc:302] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:26.351365 7759 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:57:26.351609 9505 raft_consensus.cc:2781] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:57:26.351739 9505 raft_consensus.cc:686] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [term 4 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:26.352046 9505 consensus_queue.cc:237] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:26.357105 7232 catalog_manager.cc:5484] T d3ac865cf3c044ac8ae3b0383b0b73c0 P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 3 to 4, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 4 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
I20240502 01:57:26.357825 7232 catalog_manager.cc:5484] T f8f40cb87e054bb28ea0ab7aa9db20ff P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 3 to 4, leader changed from d599c2145ef344fbbe26469d4331175d (127.14.212.194) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 4 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:26.359475 7382 leader_election.cc:288] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:26.360859 7382 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e4bb4b288a4d4fd689b750da93d2fa07"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:26.352769 7779 raft_consensus.cc:1261] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Refusing update from remote peer d599c2145ef344fbbe26469d4331175d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:57:26.363723 7779 pending_rounds.cc:76] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412: Aborting all ops after (but not including) 3
I20240502 01:57:26.363849 7775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94f99eccbe34c9ebf3a375e9230994d" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:26.364212 7775 raft_consensus.cc:3032] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:57:26.364363 7775 raft_consensus.cc:729] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:26.364751 7775 consensus_queue.cc:260] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } }
I20240502 01:57:26.365273 7775 raft_consensus.cc:3037] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.363843 7779 pending_rounds.cc:97] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:57:26.369222 7779 raft_consensus.cc:2866] T 0718691ffdc04f719ad7df57987df68c P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:57:26.370616 9824 consensus_queue.cc:1035] T 0718691ffdc04f719ad7df57987df68c P d599c2145ef344fbbe26469d4331175d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.372223 7775 raft_consensus.cc:2454] T c94f99eccbe34c9ebf3a375e9230994d P fb68cc03fe5644e0b4846a80d385c412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:26.383052 7382 raft_consensus.cc:482] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:26.383273 7382 raft_consensus.cc:3037] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.386444 7382 raft_consensus.cc:504] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:26.388264 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4bb4b288a4d4fd689b750da93d2fa07" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:26.388562 7515 raft_consensus.cc:3032] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:57:26.388684 7515 raft_consensus.cc:729] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:26.389061 7515 consensus_queue.cc:260] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:26.389472 7515 raft_consensus.cc:3037] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.393438 7515 raft_consensus.cc:2454] T e4bb4b288a4d4fd689b750da93d2fa07 P d599c2145ef344fbbe26469d4331175d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:26.394001 7313 leader_election.cc:302] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters:
I20240502 01:57:26.394405 9900 raft_consensus.cc:2781] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:57:26.394573 9900 raft_consensus.cc:686] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [term 4 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:26.394886 9900 consensus_queue.cc:237] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:26.398624 7777 raft_consensus.cc:1261] T e43e7e31842d47c59425ca63bc7182da P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Refusing update from remote peer 87e2815274bf41c4ace764c5e37599b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:57:26.399221 7382 leader_election.cc:288] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:26.399211 7645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4bb4b288a4d4fd689b750da93d2fa07" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:26.399333 9505 consensus_queue.cc:1035] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.399515 7645 raft_consensus.cc:3037] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.400476 7378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3296950eda164749ab23df9d041ff456"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:26.400709 7378 raft_consensus.cc:482] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:26.400815 7378 raft_consensus.cc:3037] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.402566 7645 raft_consensus.cc:2454] T e4bb4b288a4d4fd689b750da93d2fa07 P e8314214a0484251964e4b83a87b6786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:26.403726 7378 raft_consensus.cc:504] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.404428 7378 leader_election.cc:288] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:26.405419 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296950eda164749ab23df9d041ff456" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:26.405694 7780 raft_consensus.cc:3037] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.406180 7377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3c56e67fc824a8799e0218b245f52c9"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:26.406446 7377 raft_consensus.cc:482] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:26.406651 7377 raft_consensus.cc:3037] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.407930 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296950eda164749ab23df9d041ff456" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:26.408249 7515 raft_consensus.cc:3032] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:57:26.408380 7515 raft_consensus.cc:729] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:26.408762 7515 consensus_queue.cc:260] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.409257 7515 raft_consensus.cc:3037] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.409651 7377 raft_consensus.cc:504] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:26.411032 7377 leader_election.cc:288] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:26.411154 7514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c56e67fc824a8799e0218b245f52c9" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:26.411371 7514 raft_consensus.cc:3032] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:26.411453 7514 raft_consensus.cc:729] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:26.411761 7514 consensus_queue.cc:260] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:26.411871 7780 raft_consensus.cc:2454] T 3296950eda164749ab23df9d041ff456 P fb68cc03fe5644e0b4846a80d385c412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:26.411989 7776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c56e67fc824a8799e0218b245f52c9" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:26.412173 7514 raft_consensus.cc:3037] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.412212 7776 raft_consensus.cc:3037] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.414702 7232 catalog_manager.cc:5484] T c94f99eccbe34c9ebf3a375e9230994d P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 3 to 4, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 4 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } }
I20240502 01:57:26.415145 7776 raft_consensus.cc:2454] T d3c56e67fc824a8799e0218b245f52c9 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 3.
I20240502 01:57:26.415145 7514 raft_consensus.cc:2454] T d3c56e67fc824a8799e0218b245f52c9 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 3.
I20240502 01:57:26.415254 7232 catalog_manager.cc:5484] T e4bb4b288a4d4fd689b750da93d2fa07 P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 3 to 4, leader changed from d599c2145ef344fbbe26469d4331175d (127.14.212.194) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 4 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
I20240502 01:57:26.415684 7313 leader_election.cc:302] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters:
I20240502 01:57:26.416457 9505 raft_consensus.cc:2781] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:26.416620 9505 raft_consensus.cc:686] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5 [term 3 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:26.416930 9505 consensus_queue.cc:237] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } }
I20240502 01:57:26.418076 7515 raft_consensus.cc:2454] T 3296950eda164749ab23df9d041ff456 P d599c2145ef344fbbe26469d4331175d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:26.419368 7313 leader_election.cc:302] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters:
I20240502 01:57:26.419744 9900 raft_consensus.cc:2781] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:57:26.419926 9900 raft_consensus.cc:686] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [term 4 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:26.420205 9900 consensus_queue.cc:237] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.422848 7381 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7dad5d30968a4698abc6688c2a71cddf"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:26.423141 7381 raft_consensus.cc:482] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:26.423316 7381 raft_consensus.cc:3037] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.426926 7381 raft_consensus.cc:504] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.428239 7381 leader_election.cc:288] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:26.428407 7645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dad5d30968a4698abc6688c2a71cddf" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:26.428644 7645 raft_consensus.cc:3037] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.429860 7776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dad5d30968a4698abc6688c2a71cddf" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:26.430194 7776 raft_consensus.cc:3032] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:57:26.430325 7776 raft_consensus.cc:729] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:26.430684 7776 consensus_queue.cc:260] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.431174 7776 raft_consensus.cc:3037] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:57:26.431728 7645 raft_consensus.cc:2454] T 7dad5d30968a4698abc6688c2a71cddf P e8314214a0484251964e4b83a87b6786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:26.432508 7314 leader_election.cc:302] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:26.432863 9505 raft_consensus.cc:2781] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:57:26.433074 9505 raft_consensus.cc:686] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [term 4 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:26.433395 9505 consensus_queue.cc:237] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.438537 7228 catalog_manager.cc:5484] T 3296950eda164749ab23df9d041ff456 P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 3 to 4, leader changed from d599c2145ef344fbbe26469d4331175d (127.14.212.194) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 4 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:26.438818 7776 raft_consensus.cc:2454] T 7dad5d30968a4698abc6688c2a71cddf P fb68cc03fe5644e0b4846a80d385c412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 4.
I20240502 01:57:26.439116 7381 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7ac302b18a4a41239b479f539f590b40"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:26.439175 7228 catalog_manager.cc:5484] T 7dad5d30968a4698abc6688c2a71cddf P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 3 to 4, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 4 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:26.439391 7381 raft_consensus.cc:482] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:26.439679 7228 catalog_manager.cc:5484] T d3c56e67fc824a8799e0218b245f52c9 P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 2 to 3, leader changed from d599c2145ef344fbbe26469d4331175d (127.14.212.194) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 3 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } }
I20240502 01:57:26.440204 7515 raft_consensus.cc:1261] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Refusing update from remote peer 87e2815274bf41c4ace764c5e37599b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:57:26.440343 7515 pending_rounds.cc:76] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d: Aborting all ops after (but not including) 3
I20240502 01:57:26.440405 7515 pending_rounds.cc:97] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:57:26.440546 7515 raft_consensus.cc:2866] T e43e7e31842d47c59425ca63bc7182da P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:57:26.441428 9505 consensus_queue.cc:1035] T e43e7e31842d47c59425ca63bc7182da P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.442293 7381 raft_consensus.cc:3037] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.449471 7381 raft_consensus.cc:504] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.450368 7381 leader_election.cc:288] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:26.451841 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac302b18a4a41239b479f539f590b40" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:26.452080 7648 raft_consensus.cc:3037] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.453462 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac302b18a4a41239b479f539f590b40" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:26.453760 7515 raft_consensus.cc:3032] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:26.453842 7515 raft_consensus.cc:729] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:26.454113 7515 consensus_queue.cc:260] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.454533 7515 raft_consensus.cc:3037] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.455432 7648 raft_consensus.cc:2454] T 7ac302b18a4a41239b479f539f590b40 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 3.
I20240502 01:57:26.458935 7314 leader_election.cc:302] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:26.459529 9505 raft_consensus.cc:2781] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:26.459725 9505 raft_consensus.cc:686] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5 [term 3 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:26.460005 9505 consensus_queue.cc:237] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.462221 7377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7ca112cbaf214bd7aad56b8e1a336ddd"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:26.462436 7377 raft_consensus.cc:482] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:26.462464 7515 raft_consensus.cc:2454] T 7ac302b18a4a41239b479f539f590b40 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 3.
I20240502 01:57:26.462551 7377 raft_consensus.cc:3037] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.465546 7377 raft_consensus.cc:504] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.465983 7232 catalog_manager.cc:5484] T 7ac302b18a4a41239b479f539f590b40 P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 2 to 3, leader changed from d599c2145ef344fbbe26469d4331175d (127.14.212.194) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 3 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:26.466746 7647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca112cbaf214bd7aad56b8e1a336ddd" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:26.467005 7647 raft_consensus.cc:3037] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.467253 7377 leader_election.cc:288] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:26.468423 7377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5bcb764b10d43419e58fdcfeadb2143"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:26.468631 7377 raft_consensus.cc:482] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:26.468788 7377 raft_consensus.cc:3037] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.469575 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca112cbaf214bd7aad56b8e1a336ddd" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:26.469884 7780 raft_consensus.cc:3032] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:26.470000 7780 raft_consensus.cc:729] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:26.470018 7647 raft_consensus.cc:2454] T 7ca112cbaf214bd7aad56b8e1a336ddd P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 3.
I20240502 01:57:26.470527 7314 leader_election.cc:302] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:26.470996 7780 consensus_queue.cc:260] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.471177 9505 raft_consensus.cc:2781] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:26.471340 9505 raft_consensus.cc:686] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5 [term 3 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:26.471390 7780 raft_consensus.cc:3037] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.471654 9505 consensus_queue.cc:237] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.474289 7780 raft_consensus.cc:2454] T 7ca112cbaf214bd7aad56b8e1a336ddd P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 3.
I20240502 01:57:26.477377 7229 catalog_manager.cc:5484] T 7ca112cbaf214bd7aad56b8e1a336ddd P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 2 to 3, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 3 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:26.483657 7377 raft_consensus.cc:504] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.484477 7377 leader_election.cc:288] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Requested vote from peers fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255), d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047)
I20240502 01:57:26.485051 7776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcb764b10d43419e58fdcfeadb2143" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:26.485308 7776 raft_consensus.cc:3037] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.488312 7776 raft_consensus.cc:2454] T d5bcb764b10d43419e58fdcfeadb2143 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 3.
I20240502 01:57:26.488987 7311 leader_election.cc:302] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, fb68cc03fe5644e0b4846a80d385c412; no voters:
I20240502 01:57:26.489337 9901 raft_consensus.cc:2781] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:26.489493 9901 raft_consensus.cc:686] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5 [term 3 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:26.489780 9901 consensus_queue.cc:237] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.491920 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcb764b10d43419e58fdcfeadb2143" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:26.492174 7515 raft_consensus.cc:3032] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:26.492249 7515 raft_consensus.cc:729] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:26.492530 7515 consensus_queue.cc:260] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.493417 7515 raft_consensus.cc:3037] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.499022 7229 catalog_manager.cc:5484] T d5bcb764b10d43419e58fdcfeadb2143 P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 2 to 3, leader changed from d599c2145ef344fbbe26469d4331175d (127.14.212.194) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 3 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:26.505010 7515 raft_consensus.cc:2454] T d5bcb764b10d43419e58fdcfeadb2143 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 3.
I20240502 01:57:26.505846 7377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "04426a27d50548b9a3eb0e3eb09a5bcc"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:26.506168 7377 raft_consensus.cc:482] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:26.506351 7377 raft_consensus.cc:3037] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.509582 7377 raft_consensus.cc:504] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:26.510701 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04426a27d50548b9a3eb0e3eb09a5bcc" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d599c2145ef344fbbe26469d4331175d"
I20240502 01:57:26.510934 7515 raft_consensus.cc:3032] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:26.511018 7515 raft_consensus.cc:729] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d599c2145ef344fbbe26469d4331175d, State: Running, Role: LEADER
I20240502 01:57:26.511096 7647 raft_consensus.cc:1261] T d727fc7ec2054b46aafb44ccf4bd0ad8 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Refusing update from remote peer 87e2815274bf41c4ace764c5e37599b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:57:26.511330 7515 consensus_queue.cc:260] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:26.511720 7515 raft_consensus.cc:3037] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.511941 9505 consensus_queue.cc:1035] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.514667 7515 raft_consensus.cc:2454] T 04426a27d50548b9a3eb0e3eb09a5bcc P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 3.
I20240502 01:57:26.515132 7313 leader_election.cc:302] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, d599c2145ef344fbbe26469d4331175d; no voters:
I20240502 01:57:26.515473 9505 raft_consensus.cc:2781] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:26.515631 9505 raft_consensus.cc:686] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5 [term 3 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:26.515661 7377 leader_election.cc:288] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Requested vote from peers d599c2145ef344fbbe26469d4331175d (127.14.212.194:39047), e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355)
I20240502 01:57:26.515937 9505 consensus_queue.cc:237] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } }
I20240502 01:57:26.517613 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04426a27d50548b9a3eb0e3eb09a5bcc" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:26.517855 7648 raft_consensus.cc:3037] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.519636 7229 catalog_manager.cc:5484] T 04426a27d50548b9a3eb0e3eb09a5bcc P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 2 to 3, leader changed from d599c2145ef344fbbe26469d4331175d (127.14.212.194) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 3 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } peers { permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 } } peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } }
I20240502 01:57:26.521365 7648 raft_consensus.cc:2454] T 04426a27d50548b9a3eb0e3eb09a5bcc P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 3.
I20240502 01:57:26.526046 7515 raft_consensus.cc:1261] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: Refusing update from remote peer 87e2815274bf41c4ace764c5e37599b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:57:26.526227 7515 pending_rounds.cc:76] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d: Aborting all ops after (but not including) 3
I20240502 01:57:26.526304 7515 pending_rounds.cc:97] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:57:26.526404 7515 raft_consensus.cc:2866] T d727fc7ec2054b46aafb44ccf4bd0ad8 P d599c2145ef344fbbe26469d4331175d [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:57:26.527107 9901 consensus_queue.cc:1035] T d727fc7ec2054b46aafb44ccf4bd0ad8 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d599c2145ef344fbbe26469d4331175d" member_type: VOTER last_known_addr { host: "127.14.212.194" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:26.531764 7377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46549b89a28243a7b4245c14ecba6008"
dest_uuid: "87e2815274bf41c4ace764c5e37599b5"
from {username='slave'} at 127.0.0.1:43004
I20240502 01:57:26.532027 7377 raft_consensus.cc:482] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:26.532167 7377 raft_consensus.cc:3037] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.535389 7377 raft_consensus.cc:504] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.536823 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46549b89a28243a7b4245c14ecba6008" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e8314214a0484251964e4b83a87b6786"
I20240502 01:57:26.537101 7648 raft_consensus.cc:3037] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.540635 7648 raft_consensus.cc:2454] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 3.
I20240502 01:57:26.541252 7314 leader_election.cc:302] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e2815274bf41c4ace764c5e37599b5, e8314214a0484251964e4b83a87b6786; no voters:
I20240502 01:57:26.541637 9505 raft_consensus.cc:2781] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:57:26.541793 9505 raft_consensus.cc:686] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5 [term 3 LEADER]: Becoming Leader. State: Replica: 87e2815274bf41c4ace764c5e37599b5, State: Running, Role: LEADER
I20240502 01:57:26.542088 9505 consensus_queue.cc:237] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.544824 7377 leader_election.cc:288] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5 [CANDIDATE]: Term 3 election: Requested vote from peers e8314214a0484251964e4b83a87b6786 (127.14.212.195:35355), fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196:34255)
I20240502 01:57:26.545344 7776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46549b89a28243a7b4245c14ecba6008" candidate_uuid: "87e2815274bf41c4ace764c5e37599b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "fb68cc03fe5644e0b4846a80d385c412"
I20240502 01:57:26.545612 7776 raft_consensus.cc:3032] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:57:26.545725 7776 raft_consensus.cc:729] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fb68cc03fe5644e0b4846a80d385c412, State: Running, Role: LEADER
I20240502 01:57:26.545991 7776 consensus_queue.cc:260] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } }
I20240502 01:57:26.546389 7776 raft_consensus.cc:3037] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:57:26.546356 7232 catalog_manager.cc:5484] T 46549b89a28243a7b4245c14ecba6008 P 87e2815274bf41c4ace764c5e37599b5 reported cstate change: term changed from 2 to 3, leader changed from fb68cc03fe5644e0b4846a80d385c412 (127.14.212.196) to 87e2815274bf41c4ace764c5e37599b5 (127.14.212.193). New cstate: current_term: 3 leader_uuid: "87e2815274bf41c4ace764c5e37599b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8314214a0484251964e4b83a87b6786" member_type: VOTER last_known_addr { host: "127.14.212.195" port: 35355 } } peers { permanent_uuid: "fb68cc03fe5644e0b4846a80d385c412" member_type: VOTER last_known_addr { host: "127.14.212.196" port: 34255 } } peers { permanent_uuid: "87e2815274bf41c4ace764c5e37599b5" member_type: VOTER last_known_addr { host: "127.14.212.193" port: 40913 } } }
I20240502 01:57:26.549185 7776 raft_consensus.cc:2454] T 46549b89a28243a7b4245c14ecba6008 P fb68cc03fe5644e0b4846a80d385c412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e2815274bf41c4ace764c5e37599b5 in term 3.
I20240502 01:57:26.550474 7648 raft_consensus.cc:1226] T 46549b89a28243a7b4245c14ecba6008 P e8314214a0484251964e4b83a87b6786 [term 3 FOLLOWER]: Rejecting Update request from peer fb68cc03fe5644e0b4846a80d385c412 for earlier term 2. Current term is 3. Ops: []
I20240502 01:57:26.562742 7647 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b443becac180479abb1071e4c829fbcb"
dest_uuid: "e8314214a0484251964e4b83a87b6786"